brian/wanderer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Feature: watcher based bulk uploads (#886)

Browse Source 
* initial commit

* fix watcher.js for docker

* Update docs

* update healthcheck

* update docs

---------

Co-authored-by: Christian Beutel <>
This commit is contained in:
Flomp
2026-04-20 13:46:32 +02:00
committed by GitHub
parent 1f17a8784b
commit 871bf38112
8 changed files with 155 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docker-compose.yml
View File

@@ -76,7 +76,7 @@ services:
- wanderer
restart: unless-stopped
healthcheck:
test: ["CMD", "/curl", "--fail", "http://localhost:3000/"]
test: ["CMD", "curl", "--fail", "http://localhost:3000/"]
interval: 15s
retries: 10
start_period: 20s

15
docs/src/content/docs/use/authentication.md
View File

@@ -47,6 +47,21 @@ If your instance offers OAuth logins, the enabled providers appear in <span clas
For instructions on enabling OAuth2 providers for your own instance, see the [OAuth2 setup guide](/run/backend-configuration/oauth2/).
## API Tokens
API tokens allow external tools and automated processes to interact with your <span class="-tracking-[0.075em]">wanderer</span> account without requiring your login credentials.
:::danger
API tokens grant full access to your account. Do not share them with untrusted parties.
:::
To manage your tokens:
1. Log in to your <span class="-tracking-[0.075em]">wanderer</span> instance.
2. Navigate to **Settings** > **Account** > **API Tokens**.
3. Click **Generate new token**, provide a descriptive name and optionally an expiration date and click "Save".
4. **Copy the token immediately.** For security, it will not be shown again.
## Forgot your password?
<span class="-tracking-[0.075em]">wanderer</span> offers the option to send password reset emails in case a user forgets his password.
You can click the "Forgot password" link in the login form. After requesting the reset the user will receive an email with a unique link to reset their password.

36
docs/src/content/docs/use/import-export.md
View File

@@ -5,37 +5,43 @@ description: How to import and export trails in wanderer
## Import
<span class="-tracking-[0.075em]">wanderer</span> supports bulk uploading of trails via an auto-upload folder. A cronjob fetches all files from this folder and uploads them automatically every 15 minutes. This feature is currently only available for docker installations. If you want to replicate it in a bare metal installation you will need to create your own cronjob using the `web/cron.sh` script.
<span class="-tracking-[0.075em]">wanderer</span> supports bulk uploading of trails via an auto-upload folder. A file watcher automatically detects new files added to this directory and imports them into your library.
:::note
This feature is currently only available for Docker installations. Files added to the folder while the container is not running are ignored.
:::
:::caution
Successfully uploaded files will be deleted from the auto-upload folder.
:::
:::note
Currently only GPX files are supported.
:::
### Configuration
The following environment variables must be present in the `<span class="-tracking-[0.075em]">wanderer</span>-web` docker container and set to valid values.
#### Environment variables
The following environment variable must be present in the `wanderer-web` docker container and set to a valid volume path (see below).
| Environment Variable | Description | Default |
|----------------------|------------------------------------------------------------------------|--------------|
| -------------------- | ------------------------------ | ------------ |
| UPLOAD_FOLDER | Path to the auto-upload folder | /app/uploads |
| UPLOAD_USER | Username of the account that will be the author of the uploaded trails | |
| UPLOAD_PASSWORD | Password of the account that will be the author of the uploaded trails | |
### Volume
#### Volume
Make sure to mount the upload folder as a volume to your host system. The default `docker-compose.yml` already includes this volume. Ensure that the mapped value matches the one in the `UPLOAD_FOLDER` environment variable.
### Manually run the upload job
In case you do not want to wait until the next scheduled execution you can also run the job manually:
#### API token
The bulk upload process uses API tokens to authenticate requests and determine which user account the uploaded trails should be assigned to.
1. Create an API token: Follow the steps in the [Authentication section](/use/authentication/#api-tokens) to generate a new API token.
2. Prepare the folder structure: Create the folder: Inside your UPLOAD_FOLDER, create a sub-folder named exactly after your API token.
3. Upload: Move your trail files (e.g., .gpx, .fit, or .kml) into that sub-folder.
**Example structure**:
`/app/uploads/wanderer_key_<...>/my_trail.gpx`
```bash
docker exec -it wanderer-web run-parts /etc/periodic/15min
```
## Export
To export a single trail head over to `/trails` and select the trail you want to export. From the <span class="inline-block w-8 h-8 bg-primary rounded-full text-center text-white">⋮</span> menu select "Export". You can export the route data either in GPX or in GeoJSON format. Furthermore, you can choose whether you want to include the photos and the summit book of the trail. In any case, <span class="-tracking-[0.075em]">wanderer</span> will create a ZIP archive with all the data that is then downloaded.
To export selected trails head over to `/trails` and select the trails you want to export. From the <span class="inline-block w-8 h-8 bg-primary rounded-full text-center text-white">⋮</span> menu select "Export". You can export the route data either in GPX or in GeoJSON format. Furthermore, you can choose whether you want to include the photos and the summit book of the trail. In any case, <span class="-tracking-[0.075em]">wanderer</span> will create a ZIP archive with all the data that is then downloaded.
You can also export all of your trails at once. To do so, head over to `/settings/export` and click "Export all trails". The other steps remain analogous to exporting a single trail.

37
web/Dockerfile
View File

@@ -1,33 +1,3 @@
FROM curlimages/curl:8.18.0 AS download-env
# renovate: datasource=github-releases depName=stunnel/static-curl packageName=stunnel/static-curl
ENV CURL_VERSION=8.18.0
RUN set -eux ; \
ARCHITECTURE="$(uname -m)" ; \
case $ARCHITECTURE in \
x86_64) ARCHITECTURE="x86_64" ;; \
aarch64 | armv8* | arm64) ARCHITECTURE="aarch64" ;; \
*) \
echo "(!) Architecture $ARCHITECTURE unsupported" ; \
exit 1 \
;; \
esac ; \
curl \
--connect-timeout 10 \
--fail \
--location \
--max-time 300 \
--output /tmp/curl.tar.xz \
--proto '=https' \
--show-error \
--silent \
--tlsv1.2 \
"https://github.com/stunnel/static-curl/releases/download/${CURL_VERSION}/curl-linux-${ARCHITECTURE}-glibc-${CURL_VERSION}.tar.xz" \
; \
tar -xJf /tmp/curl.tar.xz -C /tmp ; \
chmod +x /tmp/curl ;
FROM node:22-alpine AS build-env
WORKDIR /app
@@ -45,12 +15,11 @@ FROM node:22-alpine
WORKDIR /app/uploads
WORKDIR /app
COPY --from=download-env /tmp/curl /curl
COPY --from=build-env /app /app
COPY ./cron.sh /etc/periodic/15min/cron
RUN chmod +x /etc/periodic/15min/cron
RUN apk add --no-cache curl
COPY watcher.js /app/watcher.js
CMD crond && node build
CMD ["npm", "run", "start"]
EXPOSE 3000

62
web/cron.sh
View File

@@ -1,62 +0,0 @@
#!/bin/ash
# API endpoint URL
API_URL="http://localhost:3000/api/v1"
# Folder containing files to upload
UPLOAD_FOLDER=$UPLOAD_FOLDER
# Credentials for login
USERNAME=$UPLOAD_USER
PASSWORD=$UPLOAD_PASSWORD
login() {
local username="$1"
local password="$2"
response=$(/curl -c cookie.txt --location --request POST "$API_URL/auth/login" --header 'Content-Type: application/json' --data-raw "{\"username\": \"$username\", \"password\": \"$password\"}")
# Check if login was successful (look for "200 OK" in response headers)
if [ $? -eq 0 ] && [ "$(echo "$response" | grep -c "token")" -eq 1 ]; then
echo "[INFO] [$(date +"%T")]: Login successful. Cookie obtained." > /proc/1/fd/1
else
echo "[ERROR] [$(date +"%T")]: Login failed. Unable to obtain cookie." > /proc/1/fd/1
exit 1
fi
}
# Function to upload file and delete if successful
upload_and_delete() {
local file="$1"
ls "$file"
# API call to upload file
response=$(/curl -b cookie.txt --location --request PUT "$API_URL/trail/upload" --header 'Content-Type: multipart/form-data' -F "file=@-" -F "name=$base_name" <"$file")
# Check if API call was successful (status code 200)
if [ $? -eq 0 ] && [ "$(echo "$response" | grep -c "author")" -eq 1 ]; then
echo "[INFO] [$(date +"%T")]: File $file uploaded successfully." > /proc/1/fd/1
# Delete the file
rm "$file"
echo "[INFO] [$(date +"%T")]: File $file deleted."
else
echo $response
echo "[ERROR] [$(date +"%T")]: Failed to upload file $file." > /proc/1/fd/1
fi
}
# Login to obtain cookie
if [ -n "$USERNAME" ] && [ -n "$PASSWORD" ]; then
echo "[INFO] [$(date +"%T")]: Starting auto-upload" > /proc/1/fd/1
login "$USERNAME" "$PASSWORD"
# Iterate over each file in the folder
for file in "$UPLOAD_FOLDER"/*; do
# Check if file exists and is a regular file
if [ -f "$file" ]; then
upload_and_delete "$file"
fi
done
echo "[INFO] [$(date +"%T")]: Auto-upload completed" > /proc/1/fd/1
fi

51
web/package-lock.json generated
View File

@@ -36,6 +36,7 @@
"chart.js": "^4.5.1",
"chartjs-plugin-crosshair": "^2.0.0",
"chartjs-plugin-zoom": "^2.1.0",
"chokidar": "^5.0.0",
"crypto-random-string": "^5.0.0",
"felte": "^1.3.0",
"heic2any": "^0.0.4",
@@ -2857,16 +2858,15 @@
}
},
"node_modules/chokidar": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz",
"integrity": "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==",
"dev": true,
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-5.0.0.tgz",
"integrity": "sha512-TQMmc3w+5AxjpL8iIiwebF73dRDF4fBIieAqGn9RGCWaEVwQ6Fb2cGe31Yns0RRIzii5goJ1Y7xbMwo1TxMplw==",
"license": "MIT",
"dependencies": {
"readdirp": "^4.0.1"
"readdirp": "^5.0.0"
},
"engines": {
"node": ">= 14.16.0"
"node": ">= 20.19.0"
},
"funding": {
"url": "https://paulmillr.com/funding/"
@@ -5120,13 +5120,12 @@
"license": "MIT"
},
"node_modules/readdirp": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-4.1.1.tgz",
"integrity": "sha512-h80JrZu/MHUZCyHu5ciuoI0+WxsCxzxJTILn6Fs8rxSnFPh+UVHYfeIxK1nVGugMqkfC4vJcBOYbkfkwYK0+gw==",
"dev": true,
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-5.0.0.tgz",
"integrity": "sha512-9u/XQ1pvrQtYyMpZe7DXKv2p5CNvyVwzUB6uhLAnQwHMSgKMBR62lc7AHljaeteeHXn11XTAaLLUVZYVZyuRBQ==",
"license": "MIT",
"engines": {
"node": ">= 14.18.0"
"node": ">= 20.19.0"
},
"funding": {
"type": "individual",
@@ -5553,6 +5552,36 @@
"typescript": ">=5.0.0"
}
},
"node_modules/svelte-check/node_modules/chokidar": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz",
"integrity": "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==",
"dev": true,
"license": "MIT",
"dependencies": {
"readdirp": "^4.0.1"
},
"engines": {
"node": ">= 14.16.0"
},
"funding": {
"url": "https://paulmillr.com/funding/"
}
},
"node_modules/svelte-check/node_modules/readdirp": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-4.1.2.tgz",
"integrity": "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 14.18.0"
},
"funding": {
"type": "individual",
"url": "https://paulmillr.com/funding/"
}
},
"node_modules/svelte-i18n": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/svelte-i18n/-/svelte-i18n-4.0.1.tgz",

2
web/package.json
View File

@@ -5,6 +5,7 @@
"scripts": {
"dev": "vite dev",
"build": "vite build",
"start": "node watcher.js & node build",
"preview": "vite preview",
"test": "npm run test:integration && npm run test:unit",
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json",
@@ -58,6 +59,7 @@
"chart.js": "^4.5.1",
"chartjs-plugin-crosshair": "^2.0.0",
"chartjs-plugin-zoom": "^2.1.0",
"chokidar": "^5.0.0",
"crypto-random-string": "^5.0.0",
"felte": "^1.3.0",
"heic2any": "^0.0.4",

70
web/watcher.js Normal file
View File

@@ -0,0 +1,70 @@
import fs from 'node:fs';
import path from 'node:path';
import chokidar from 'chokidar';
const ORIGIN = "http://localhost:3000";
const { UPLOAD_FOLDER } = process.env;
if (UPLOAD_FOLDER) {
const uploadPath = path.resolve(UPLOAD_FOLDER);
// Ensure directory exists
if (!fs.existsSync(uploadPath)) {
try {
fs.mkdirSync(uploadPath, { recursive: true });
} catch (err) {
console.error(`[File Watcher] Failed to create directory ${uploadPath}:`, err.message);
}
}
console.log(`[File Watcher] Service active. Watching: ${uploadPath}`);
const watcher = chokidar.watch(uploadPath, {
persistent: true,
ignoreInitial: true,
awaitWriteFinish: {
stabilityThreshold: 2000,
pollInterval: 100
}
});
watcher.on('add', async (filePath) => {
const relative = path.relative(uploadPath, filePath);
const [token] = relative.split(path.sep);
if (!token || token === '.' || path.basename(filePath).startsWith('.')) return;
try {
const fileBuffer = fs.readFileSync(filePath);
const fileBlob = new Blob([fileBuffer]);
const formData = new FormData();
formData.append('file', fileBlob, path.basename(filePath));
formData.append('ignoreDuplicates', "true");
const response = await fetch(`${ORIGIN}/api/v1/trail/upload`, {
method: 'PUT',
headers: { 'Authorization': `Bearer ${token}` },
body: formData
});
if (response.ok) {
fs.unlinkSync(filePath);
console.log(`[File Watcher] Uploaded and removed: ${path.basename(filePath)}`);
} else {
const errorText = await response.text();
console.error(`[File Watcher] Server rejected ${path.basename(filePath)} (${response.status}): ${errorText}`);
}
} catch (err) {
console.error(`[File Watcher] Upload error:`, err.message);
}
});
watcher.on('error', error => console.error(`[File Watcher] Watcher error: ${error}`));
process.on('SIGTERM', () => {
watcher.close();
});
} else {
console.log('[File Watcher] Disabled: UPLOAD_FOLDER not provided.');
}