Is it possible to get maps in an easier way than MapBox?
I tried to make a small map of my area at the tip of Norway, from Kristiansand to Lyngdal (which is as far as I can imagine going even when I retire in a few years) on MapBox, but I gotta admit that I gave up. The learning curve is steep. I wanted to use TukTuk in SignalK to have an extra, fully offline map available on the phone/tablet in addition to the one on the Garmin 720 in the (open) wheelhouse, to know exactly where the depth changes are when setting crawfish and crab pots. But is there a simpler way to get what I need than MapBox?
240 Replies
you could try opencpn & download mbtiles with sasplanet. On android only though. Sasplanet has navionics and satellite images which it can save in a format opencpn can read. Sat images are great!! π
Thanks, but I'm not looking for something to work only one device. I need it on the SignalK server, so it's accessable on all my devices, which includes a Windows Surface tablet/laptop.
You can create mbtiles using sasplanet and use the mbtiles using signalk/freeboard possibly will work with tuktuk as well. It is a nice solution especially if you interested in a map for a small region
You should not use the SAS.Planet...
Three steps:
1) Install GaladrielCache on your signalk computer
2) Configure charts-plugin to use GaladrielCache
3) Explore your region on the map via TukTuk (or freeboard-sk or GaladrielMapSK)
After some time you will have the required map for off-line use.
@Ba2 The tiles work for Tuktuk as well, yeah. The reason I want to use that is that I liked the way it showed overlays fit me better.
@ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ Thanks! I'll have a look at that. I will do a full image backup of the Pi first, and then install it. Maybe I can find the cahce too and save that on my SSD.
Having looked at it now I'm afraid that is a bit over my level, I don't understand how to install and use it.
Thanks! I need to find out how to install php and Apache, I have never used anything like that.
apt install apache2 php π
But need PHP7, not PHP8.Thanks! I'll try!
I think I have set it up according to the instructions (except for the TOR connection, which I don't have, I don't think I need that since I won't be using AIS), but I get "The requested URL not found", It's probably me being thick, I have tried with http://127.0.0.0/galadriel-map, http://127.0.0.0/galadrielmap and http://127.0.0.0/map but none of these work.
If you are on the same computer, then:
http://127.0.0.1/map/
It is a GaladrielMap chartplotter.Fantastic, thank you very much! It works! Now the final question, where do I find the cached map tiles? I looked around, but didn't see anything. I have not made a special card for it, so there is nothing as sdb1 at the moment. Do I need to use that, or will the map tiles end up another place?
/GaladrielMap/tileproxy/tiles/
but if you want to configure Signal K Charts plugin for use this cache, you must create 'Online chart provider' and set URL to http://127.0.0.1/tileproxy/tiles.php?z={z}&x={x}&y={y}&r=OpenTopoMap for OpenTopo, for example.
Then all features of the GaladrielCache will be available from any SignalK chartplotter.Great, thanks again!
I see that loading the maps is very slow, especially the NAIS Norwegian that I would like to use. But I'll just let it go until it's down.
This may be a mass download ban. I haven't checked. To overcome - you need TOR π
BTW, for Navionics you must get a fresh
/GaladrielMap/tileproxy/mapsources/ files from fresh zip.Thanks, but I don't need Navionics, I am good with the Norwegian maps, those are the ones I'm used to. And I do have a NordVPN account. Maybe that will help? π€
I think I'm going to set up a virtual Debian on my server and run it from there to see if that helps. π
It didn't, so now I have installed Tor. I just need to find out how I connect to it. This is something I've never used before.
And using the file that came with Galadriel-map, it says "Tor quit during startup". I have added the data lines from the torrc file that was created by the system at the top of the file in the zip, but nothing. Sorry if I'm stupid, but I just can't get it working.
Also how can I see in GaladrielMap that it's going via Tor? Or do I do that by using the Tor browser to connect to Galadriel Map?
You should just replace the original torrc with a torrc from the zip.
The torrc from zip should work, but it's better to comment out the lines "UseBridges" and "Bridge" at the bottom of file if the TOR is not blocked in your country.
The GaladrielMap suite uses the TOR for two purposes:
1) As transport for netAIS
2) As proxy for GaladrielCache. The idea is to change the "exit node" every time after downloading several tiles.
The bad news is that this mechanism is disabled everywhere. To enable it, you need to edit the map source description file.
The second bad news is that such a mechanism is not implemented for NAIS Norwegian map.
I can add the required to map source description file for NAIS Norwegian map tomorrow.
Done. See new https://github.com/VladimirKalachikhin/Galadriel-map/blob/master/emergencykit/GaladrielMap-2.9_GaladrielCache-2_gpsdPROXY-0_netAIS-1_NoVectorTiles_.zip
I improved the map source description file for NAIS Norwegian map. It works fine for me without TOR proxy. But the ability to use a TOR proxy has been added too.
Thank you so very much for your patience and help! Now it floats like crazy here! I am afraid I will go all completist and try to cache the whole coast of Norway! π€£
@ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ One question that I stubmled over today: When i move this over to the boat, from the virtual Debian I'm using to get the map tiles, I would like to move the whole GaladrielMap directory to my SSD, to make it independent of the OS on the micro SD. I edited the file apache2_galadrielmap.conf and replaced all instances of with but I get (I use port 8080 to allow SignalK to keep port 80, but that is not a problem, at least not on the virtual Debian). And I have edited it one place in params.php. Is there another file where I need to edit the path?
I see that the error message refers to the original directory: So there has to be somewhere I haven't changed it yet. I have changed the directory statements in the apache2_galadrielmap.conf file, but left the alias, at least I think that's correct. Like this:
Thanks! I tried that version first, using the full directory on all the aliases, but that didn't work either.
Same error message, "Forbidden". I have checked the rights to the directory /media/pi/Docker/GaladrielMap and even changed it to allow anybody to change it, but to no avail.
Server configuration is magic.
The rights to the directory are not enough, you need the rights to the files:
chmod -R ...Yeah, and I'm afraid I'm no magician. I followed the chown and chmod in the instructions again, this time with the -R swithc, like this:
Still the same "Forbidden"'. π’ But now I think I'm closer, in the Apache log I see: I'm sure that tells you something, even if I don't understand it.
I tried and that shows that most are while a few are Shouldn't they all be the same after the chmod?
I also tried after a tip on search, but nothing. That made it worse.
You set
chown -hR :www-data /media/pi/Docker/GaladrielMap, but not a chown -hR :www-data /media.
User www-data has no permission to /media/pi/Docker. But this path part is present in the Directory option.Aha, I see! I think the setup har a bit too many directories to keep track of. I think I will try to use the original directory, but try to mount a separate partition on the SSD to that directory. Thanks!
You can set a "ServerRoot" directory, and use relative paths.
Ah, thanks! That sounds better!
Hm... Still not working, and I have done the chown again recursively for GaladrielMap. I have set the directories to , and I have changed both document root in sites-available/000-default.conf and Directory in apache2.conf But I still get the Forbidden. I tried too, but with the same result. Sorry if I'm dense, I just haven't done this at all before...
"ServerRoot" is server option, not directory. It is located in the global
apache2.conf. But it's not fashionable to use it now.
It is possible not to configure "Alias", just symlink GaladrielMap/map/ dir as map/ to default Apache www dir (/var/www/?, set in apache2.conf)
The www path is not a filesystem path, so common way:
1) give the permissions to the filesystem path
2) describe www path as "Alias" or set www path === file system pathA separate partition for" GaladrielMap" mounted to root turned out to be the smoothest solution. Nothing to change except for mounting it in fstab. Thanks again for your hjelp! π
It was invented for a router where there is no "disk", but there is a SD slot. But this is a simple and universal solution.
I have one problem left, it seems. I have GPSD on a separate pi, with the address 192.168.3.2:2947. Is it possible to add that somewhere? From what I can see in the params.php there's only an option for local GPSD.
/GaladrielMap/map/gpsdPROXY/params.php
$dataSourceHost
$dataSourcePort
Just the local gpsd is a default.Thank you! I changed that, but I can't see my boat. Is there something else I shoould change? I have put in this:
The "//" is comment. You must:
My bad, I keep expecting the # comment from Python or -- from LUA... But I'm afraid it's still not working. Is there something else I should try? And is there a error log somewhere that I can look at and maybe find out why it doesn't connect? I know the addresses and ports are correct, I am using GPSD both in SignalK, Homeassistant and Node-RED.
Heh, you set $dataSourceHost twice, and the second time - on SignalK which the GNSS receiver is not connected to, as far as I understand.
Of course I had... So now I have removed that, but still no fun. Should I edit anything in ?
No.
Is your gpsd on 192.168.3.2:2947? Check it by
telnet 192.168.3.2 2947. Must be {"class":"VERSION","release":"3.22","rev":"3.22","proto_major":3,"proto_minor":14}
Has the gpsdPROXY been restarted after changing the config? Try pkill -f gpsdPROXY.php and reload GaladrielMap window in browser. Do ps -A | grep gpsd. Is gpsdPROXY run?Theres https://map.openseamap.org/ but they are very map not chart
Works with leaflet if youre just trying to visualise things rather than nav
@ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ I get the correct answer on Telnet from GPSd. The whole Pi was restarted. The pkill did not change anything. But gives nothing, so I think it's not running. @JonR I'm afraid that's useless for me. I need precise depth curves down to at least 200 meters for setting my crawfish pots (I'm a very eager hobby fisher), and as far as I know only Norwegian charts gives me that.
Ok,
You can see some info or error.
So it says it's running, but there is nothing on grep.
π It's ok. π see
sudo crontab -u www-data -l
The gpsdPROXY cannot connect to the gpsd. Check the /GaladrielMap/map/gpsdPROXY/params.phpShould the crontab command give a result? It gave nothing. And I have attached the full contents of the file here. π
It should work...
Maybe I'm missing something. So, you open http://localhost/map/? And you see the map, but you don't see your position?
Actually I'm opening http://192.168.3.1:8080/map/ (the address of the boat's Pi, and 8080 because I have SignalK on port 80).
And I see the map, but no position.
(Oh, for the moment I'm opening it from another network segment, I have the 3.x segment routed via my Server, the main segment is 2.x.)
But http://localhost:8080/map/ doesn't give me the position either. Is it because I'm using the 8080 port for the map, so that has to be changed somewhere else too?
No.
Open http://192.168.3.1:8080/map/ in the browser, and open the browser console (Ctrl-Shift-i). See about "spatialWebSocket".
Does the
telnet 192.168.3.2 2947 from the 192.168.3.1 connect to gpsd?Yes, I telneted from 3.1 to 3.2 in the test above. Same result now:
Heh. http is tricky.
Replace the line
if($gpsdProxyHost=='localhost' or $gpsdProxyHost=='127.0.0.1' or $gpsdProxyHost=='0.0.0.0') $gpsdProxyHost = $_SERVER['HTTP_HOST'];
with the line
if($gpsdProxyHost=='localhost' or $gpsdProxyHost=='127.0.0.1' or $gpsdProxyHost=='0.0.0.0') $gpsdProxyHost = $_SERVER['SERVER_NAME'];
in the /GaladrielMap/map/index.html file.
This is line number 1155
Oh, sorry: in /GaladrielMap/map/index.php file.Bingo! You have a winner! π π π That works, thank you very much! Btw can I have NAIS as the default map instead of OpenTopo? I always start up in the middle of Moscow with OpenTopo when I open. π Even though it now moves to my actual position within a second or two, I still have OpenTopo.
π¦ Must be a last map(s) and a last position.
I see. OK, no biggie, the important stuff is fixed now! Thank you again!
The last map and the last position is saved in the browser cookies. Nothing special. Maybe you don't allow cookies?
That is very possible, I have strict privacy enforced. I'll make sure to whitelist the map!
These are my fishing grounds, and the chart is correct almost on the meter:
The picture above is I just happened to find another Norwegian map.
Hi again, @ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ ! Suddenly I can't download any more of the NAIS map. Can you please check if it works for you? I have tried both with and without Tor, and it doesn't change anything.
Heh. HTTP/1.1 500 Internal server error.
In the official (https://status.geonorge.no/cache.html) is same.
But the Russian hackers know the way. Be patient.
By the way
Checking tile availability:
php tilefromsource.php -z15 -x17360 -y9532 -rNAIS --maxTry=15 --checkonly
Checking all maps availability:
php checkSources.php
, see checkSources.log, use cron.
Prepare map source file to checkSources.php:
$trueTile=...
Read the docks, they are rules.
As far as I understand, the cache service of geonorge.no is broken. Forever or for a while, I do not know.
I can give you the same map from the original(?) server, but it's slower and the tiles are 1.5 times larger. Or wait for a while, suddenly it will work.Thank you! That was annoying, maybe something in an update messed it up? I will wait a few weeks, and I'll get back to this thread if it's not in by then.
Oh, I went to the website, and it says that there is a new version of UML model (whatever that means) because they have excluded "GrunnGass/pockmarks" (whatever that is). That probably means that they won't be back. Can the original server stuff be mixed with what I already have downloaded, or do I need to download everything again for that?
Yes, it looks like it.
Here are two other sources for this map. It seems to be the same visually, but files is different. Both have larger files.
In the NAIS1 the map is called "sjokartraster2", in the kartverket.no_sjokartraster - "sjokartraster". Perhaps NAIS1 is a new map, and kartverket.no_sjokartraster is a copy of the old.
I think you need to make sure which map is visually "the same", and then rename the corresponding file to NAIS.php
Or create symlink from
tiles/NAIS1/ or tiles/kartverket.no_sjokartraster/ to tiles/NAIS/Thank you very much, at least NAIS1 works, and it seems like it can work with themap I've already downloaded ! π Its not slow either. But one question: Is it possible to make Galadriel map download and cache the whole NAIS1 map in a way? Sort of pretend I'm looking at all the whole map in all zoom levels? Perhaps with the tilefromsource script in some way?
it seems like it can work with themap I've already downloadedNo. If you have not taken any action, then
tiles/NAIS and tiles/NAIS1 is different directory.
Is it possible to make Galadriel map download and cache the whole NAIS1 map in a way?"Download" menu item.
I copied all the files from NAIS to NAIS1, and they seem to load as they should even if NAIS1 has another format. And with the Download I thought I had to scroll over the map in the correct zoom levels. Or have I not found the correct numbers to add to the empty box?
I copied all the files from NAIS to NAIS1, and they seem to load as they should even if NAIS1 has another format.It's the way... But raster tiles is the raster tiles: it just 255x255 png.
And with the Download I thought I had to scroll over the map in the correct zoom levels. Or have I not found the correct numbers to add to the empty box?Dowloader load tiles from the specified zoom to max zoom (16 by default). The common use is to specify the tiles covering the required area at a suitable viewing zoom (9, for example)(easy by tap on number). Then all tiles from this zoom to the max zoom will be downloaded.
Got it, thanks! π π
Something weird is stopping me from downloading, maybe it's just me being stupid again. But I don't seem to get any maps in nais1 or even C-MAP when I try to download by selecting parts of the map, only when zooming in. Is there some kind of download queue that I may have blocked with something, that I should try to clean?
Does nothing change at all? Is that stays 0%?
Yeah, on download. The map parts only actually downloads when I zoom in on them.
Ok, try on server
cd /GaladrielMap/tileproxy
php loader.php
and look what's going on.It is ok.
What it shows
ls /GaladrielMap/tileproxy/loaderJobs
ls /GaladrielMap/tileproxy/loaderJobs/inWork
?Something is strange...
cd /GaladrielMap/tileproxy
php loaderSched.php
?
ps w -A | grep php ?Replace file
/GaladrielMap/tileproxy/loader.php with the file from the archive and try php loader.php.Do
chmod 775 /GaladrielMap/tileproxy/loaderJobsDone. And restarted Apache 2. This is the result: So it's half the size now.
ps w -A | grep loaderSched ?ls /GaladrielMap/tileproxy/loaderJobs
ls /GaladrielMap/tileproxy/loaderJobs/inWork
?We need to see what the loaderSched does, but there is a problem: the loaderSched can't just be killed, it is restarted by cron.
So you have to kill the loaderSched in one terminal and quickly start it in another terminal.
In the 1-st terminal type:
sudo kill 1497
In the 2-st terminal
cd /GaladrielMap/tileproxy
and type
php loaderSched.php
After it press Enter in 1-st terminal, and, after killing, quickly press Enter in 2-st.Do
chmod -R 775 /GaladrielMap/tileproxy/loaderJobs
πGreat, it is making progress, thanks! π I first got fooled by the percentages to think that it had stopped, but I see now that I have to close the download tab and reopen it for the numbers to update.
There is no need to keep updating percentage - it's not really important information.
However, as I recall, you seem to have set the correct permissions on those directories?
I did follow the instructions all along, so something has changed the permissions later, but I have no idea what that can be.
By the way: Do you have any desire to make a Norwegian localization of the program?
I could try. Is it many lines of text?
I hope I don't have to look through the code to find the lines to translate from English to Norwegian, that will be a disaster since I am no good at this coding.
About 170 lines π
It;s easy:
cp /GaladrielMap/map/internationalisation/en.php /GaladrielMap/map/internationalisation/no.php
and translate all or some phrases to Norwegian.I will have a look at it. π
OK, looked at it. π I think that should be doable, yeah. Just to make sure, in lines with an = sign I translate what's within single quotes after that sign, right? And in lines without that = I also translate what's within the single quotes?
Yes, all phrases in single quotes.
If your browser is not forced to use English - you will see the result immediately after reload page.
OK, I will look at it one of the first days. π
Weird... I did a few translations as a test, but I still only get the English wording on the download page. I have not have a set language, and it is the same from several devices with different browsers.
On the GaladrielMap page: right click -> View Page Source
On the opened "view-source" page: line 2 must be '<html lang="no">'
Is that true?
No, the line is "en" for some reason, on both Chrome, Firefox and Vivaldi. And that goes for both the "download" installation that I have on a virtual machine on my server and the installation on the Pi in the boat.
Ok, on a GaladrielMap page press Ctrl-Shift I, select Console and then reload the page.
Find in Console (somewhere at the top) line like this:
GET http://192.168.10.10/map/
Open dropdown.
Find "Request headers" and show the line "Accept-Language"
I don't see any of that on reload. Only error messages:
Instead the "Default levels" select "Verbose" on dropdown.
Well, the error is clear to me.
You will have to install a fresh version later.
Heh, I already fixed it? Which version are you using?
Sorry, I have had some problems, but I'm back again now. I will install the newest version this evening and see if that fixes it. Should I replace everything (and fear another permissions problem), or is there a specific file I can replace?
It's easier to update everything with the
unzip GaladrielMap-2.10_GaladrielCache-2.9_gpsdPROXY-0_netAIS-1.zip "GaladrielMap/*" -d /
But! Keep the params.php files if you have changed them.I had to set up the virtual Debian again, something went wrong with the partition. I had it working again, but then I installed the lastest version and tried to open the map, I was stuck at a zoom of 3000 and the zoom buttons were greyed out, no matter what map I tried.
Which map are you looking at?
Are there the same problems with another map?
Yeah, seems to be all maps. This is CMap:
Without any map loaded the zoom buttons work, but if I zoom in to 500 and then load a map it jumps right back to 3000 and greyed out buttons.
Turn off ALL maps.
Wait 20 sec.
Reload page.
Turn any map on.
But actually, I couldn't reproduce it.
Nope, that didn't change anything. Weird. I will copy back the setup I had there before and try to upgrade once again.
Ok.
Turn off ALL maps.
Wait 20 sec.
Reload page.
Turn on the "OpenTopoMap".
Tried it again now, and I did exactly what you say over, but same result. 3000 km zoom and greyed out buttons. Can it be related to having the partition for Galadriel Map with the NTFS filesystem? I discovered that ext4 on a VMWare virtual machine got filled up with the huge number of small PNG files.
(I have a separate partition for GaladrielMap, mounted as /GaladrielMap btw, just like I have had all the time.) Another thing that can be different is that I have not installed Tor yet on that newly restored VM. I will do that now.
Show me what's in the browser console (Ctrl-Shift-I) when the page loads?
OK, the problem was not having Tor installed. It works now, with Tor installed and running. I'll leave it running until tomorrow afternoon to see if it downloads map tiles om nais1 as it should. I'll get beack to you then.
OK, tried now, I still do not see it. This is with verbose and all other levels activated:
Scratch that, I found it in Firefox. This is what you're looking for, I think:
I tried to change the filename to nb-NO.php, but that didn't help.
Just
nb.phpThere we go! I now have my translated words. I'll do most of that tomorrow, I think. π
Btw there's some Russian in the English file. Like here, what do those words mean?
This is for the international crew. Or local rescuers... I'm not sure it's necessary at all.
Open the
ru.php and look at the corresponding variable. There's an English text.Are you still here? I heard that Russia has blocked Discord. But with your knowledge of TOR that's probably not a problem. π I have translated the file, I will read proof on the translation tomorrow. Wing in ground confused me, I had never heard about such a thing! Turns out that it's called "ekranoplan" in Norwegian, like it is in Russian, I believe. And I kept the extra text on course and position in English, as it is in the Russian file, that seemed most logical to me.
Via TOR it works, but TOR is blocked too. π The world has long been mad...
Yes, "Wing in ground" is strange, and yes, it is "ekranoplan" in Russian.
I'm going nuts here... Suddenly the latest version does not work at all! I have GaladrielMap on a separate virtual disk, mounted as /GaladrielMap. When I try the virtual disk that I had before the last version, it works. But when I try with the virtual disk with the latest version I either get "internal server error 500" or a blank page. And in that previous version the translation does not show. But it's not the translation, it doesen't work without the nb.php file either.
So I don't change anything in the rest of the system, I just replace one virtual disk with another. Can you please tell me where to look for the problem? Is it possible that it's an update for Tor that has broken it? There has been a couple the last days.
/var/log/apache2/error.log ?PHP Parse error: syntax error, unexpected '','(T_CONSTANT_ENCAPSED_STRING), expecting ')' in /GaladrielMap/map/internationalisation/nb.php on line 95
Extra quotation in the line 94 of the nb.php?You're right, of course! FIxed. I will have the Norwegian file ready in less than an hour. π
I think I must wait until tomorrow, I need to check if there is an official version of the AIS codes.
Got that, now the last detail: "Loader runs" and "loader not runs" (I'd suggest changing that to "loader running" and "loader not running" in Eglish) , is that the same as the downloader or something else?
Btw I don't hink I've said it, but I work as a translator and have been for 35 years, so it should be pretty correct. π
Oh, another thing: What file do I change to make the default map nais1 instead of Opentopo, and the default position to my position? When I open on a new device (and I have lots of devices...) I always start up on Opentopo in the middle of Mosocw.
Got that, now the last detail: "Loader runs" and "loader not runs" (I'd suggest changing that to "loader running" and "loader not running" in Eglish) , is that the same as the downloader or something else?Yes, it's Downloader status, but... I found that this variable is not used anywhere else. I have to think about whether it's right. And yes, my English is very bad π¦
Oh, another thing: What file do I change to make the default map nais1 instead of Opentopo, and the default position to my position?You have already asked: you need to allow cookies in browsers.
Your English is far from bad, actually. π I know I asked about a similar question, this is actually different. I want to have the nais1 as the default when I open the map on a new device that hasn't opened the map before.
I want to have the nais1 as the default when I open the map on a new device that hasn't opened the map before.Ah! I understood. I didn't think anyone would need it. It's hardcoded in the index.php in
var defaultMap = 'OpenTopoMap';
string (# 575); and
if(! startCenter) startCenter = L.latLng([55.754,37.62]);
string (# 583).
But it's probably better to to have it in the params.php I will do this in the next version.
Here it isIt works π
Great, I'm glad it works! I have my VM downloading everything on the NAIS1 map from zoom 6 to 16. It's using 80-90 % of the CPU on the VM! π And thanks, I'll wait until you have another version out, then. It's not something that I'm in a hurry to get.
I have my VM downloading everything on the NAIS1 map from zoom 6 to 16. It's using 80-90 % of the CPU on the VM!
/GaladrielMap/tileproxy/params.php :
$maxLoaderRuns = 15; // simultaneously working loader tasks.
Set it to 10 for less using CPU.Hi again! I have another question: How do I stop the loader jobs and restart them later? Stopping apache2 does not stop them, and disabling apache2 on startup does not block them from startup on a reboot. I would like to control this, but I can't find out how. π
This is what I mcgyverd together and it works great.
Followed https://digidem.notion.site/Preparing-and-Adding-Custom-Basemaps-to-Mapeo-b4f13019f0b842ce9315c6097f08ce36#b125e5b6233045448bac5fd28197f03e
1. Debian Bookwork (12)
2. install base and deps
apt install gdal-bin htop vim iotop npm python3-pip python3-full
npm install -g tl
npm fund
npm install -g @mapbox/mbtiles tilelive-file tilelive-http @mapbox/tilelive @mapbox/tilejson
npm fund
npm install -g tl tilelive-http tilelive-file
npm fund
more /usr/share/doc/python3.11/README.venv
mkdir -p ~/.venvs
python3 -m venv ~/.venvs/mbutil
~/.venvs/mbutil/bin/python -m pip install mbutil
3. Generate Charts
sample
tl copy -z 12 -Z 18 -b "-75.6 -1.6 -74.05 -0.275" http://ecn.t0.tiles.virtualearth.net/tiles/a{q}.jpeg?g=5162 file://./siekopai-peru-bing-tiles?filetype=jpg
tl copy -z 8 -Z 12 -b "133.896887 6.818803 134.764801 7.8" http://ecn.t0.tiles.virtualearth.net/tiles/a{q}.jpeg?g=5162 file://./palau-bing-tiles?filetype=jpg
tl copy -z 15 -Z 15 -b "133.896887 6.818803 134.764801 7.8" http://ecn.t0.tiles.virtualearth.net/tiles/a{q}.jpeg?g=5162 file://./palau-bing-tiles?filetype=jpg
- Where the -z and -Z refer to the lowest and highest zoom levels downloaded,
- The coordinates of the bounding box (w, s, e, n, in WGS 1984 unprojected).
- The file://./ is followed by the file name you want it to have
cd palau-bing-tiles
rm unwanted zoom levels in the subdir
mv metadata.json metadata.orig
/root/.venvs/mbutil/bin/mb-util --image_format=jpg /root/palau-bing-tiles/ /root/palau.mbtiles
gdalinfo palau.mbtiles
--> Test in OpenCPN
NOTES:
Bruce usually uses z8 till Z18 without Z17 (he uses: Z8 Z10 Z12 Z15 Z18)
hHanks, but I have a very good solution for the Norwegian NAIS map, and I have around 80 gig of that downloaded, so I think I'll stick to GaladrielMap. I just need to know how to pause the download.
How do I stop the loader jobs and restart them later?Tap to green "lamp" on the Loader tab to stop, and to red - to start.
Aha, thanks! I never thought about using the GUI/webpage!
Heh, sorry, NO!
Its do not pause, its completely stops!
Generally speaking, there is no way to suspend it. Everything is done there so that it downloads anyway.
You can
kill loaderSched.php and all loader.php, and after start loaderSched.php for resume.
Sorry again: It is impossible to kill loaderSched.php. Need first remove it from cron.
But if you create job file and start download from GaladrielMap - the copy of the job file saved in tileproxy/loaderJobs/oldJobs/.
You can start the download again - only missing tiles will be downloaded.I figured I could just comment out the cron job and made www-data an account that could log in. But as that user the command shows an empty file. I have set the correct permissions for the directory /var/www and a password that the account accepted, and I'm able to choose nano as the editor, it says that "No modification made", but no error messages. Any idea why? Is the cron jobitself running from another user, and running the php stuff as the www-data user?
Do you really need to pause the download? If you turn off the computer and then turn it on, the download will continue. If the download is really loading your computer - reduce the number of download threads by edit $maxLoaderRuns in
tileproxy/params.php
Another way to pause the download is remove and backup job files from tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/ Then the loader will kill itself.
To resume loading return job files to tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/ and start loaderSched.php from GaladrielMap or by tileproxy/startLoaderDaemon or just php loaderSched.phpThe problem is that even with only 4 threads it is so heavy that it kills my Pi's connection from the boat to the cabin when downloading CMap. But I guess I can move this download as well to something inside the cabin, another Pi or virtual Debian . The virtual Debian is busy with the NAIS map, which goes around 1 % every two days, up to 12 now. I know it's silly to want the full map of Norway, but I plan to do a bit more long distance traveling in a few years, when I retire.
Something strange... There is no difference which map to download, except maybe Transas.
Which specific map is causing the problem?
It's not the difference in the map, it's that I was downloading CMap directly on the Pi in the boat (while NAI1S was downloaded on a virtual Debian on the server). But yesterdayafternoon I moved that to a pi in the house with LAN instead of WiFi, and that seems to have solved it.
The problem is that even with only 4 threads it is so heavy that it kills my Pi's connection from the boat to the cabin when downloading CMap.If you mean C-MAP_raster1.php, it has a bug. Use the attached one.
I see! Thank you! π
I'm going crazy... I had to move the whole GaladrielMap directory to another virtual disk because the one I had was full, and something weird has happened with the permissions or something else. I did run afterwards and but it doesn't start downloading again. In the map's download page it's stuck on "download started", and the bullet is grey, not green or red. Trying to run the commands that you told me (killing the process and starting it again) I get this error, which is a bit different from the one I had earlier with the wrong permission:
Can you please tell me what's happening there? I checked the owner and permissions on the original VM, and it seems like they are the same..
php loaderSched.php from what user? Is he www-data or in the www-data group?
And so - the scheduler started, it started the loaders....I don't know, really. But it was done from the regular user, so I guess not. I used the stuff earlier up from you:
We need to see what the loaderSched does, but there is a problem: the loaderSched can't just be killed, it is restarted by cron. So you have to kill the loaderSched in one terminal and quickly start it in another terminal. In the 1-st terminal type:sudo kill 1497In the 2-st terminalcd /GaladrielMap/tileproxyand typephp loaderSched.phpAfter it press Enter in 1-st terminal, and, after killing, quickly press Enter in 2-st.
Heh, there are some bugs there, but I don't think it's about them π
See to
tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/.
If you see tileproxy/loaderJobs/chartName.Zoom and do not see correspondent tileproxy/loaderJobs/inWork/chartName.Zoom file - you have a problem.
Maybe you need to set write permissions to tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/ for everyone.I have this, and the directories and files have write permissions for everyone. I have been running which took more than 24 hours with all the map tiles in there. Can it be something about all those lock files?
The .lock files is PID of loaders. Currently running or, may be, dead.
.slock is a PID of scheduler.
The scheduler will delete the outdated .lock files.
The grey download indicator means that no are loader jobs and the loader is not running.
Weird. The nais1.14 file in contains 1,3 GiB with stuff to download.
I selected some stuff I had already downloaded and clicked on the "Add to download" button, and it turned green. And it is still green. Maybe that kickstarted in in a way? The download percentage is a bit weird, though... π€ But I'm pretty sure it has started, the CPU usage has gone up to where it was while downloading. I'' leave it over night and see what it says on the percentage tomorrow.
By "Add to download" you start download from www-data user.
The negative percentage mean that
tileproxy/loaderJobs/inWork/nais1.14 is bigger than tileproxy/loaderJobs/nais1.14. That is, these are different files.As I tried to say with the screen shot, which shows all files in the directory loaderJobs there is no file in loaderJobs, only in loaderJobs/inWork. The only file with the same name is in /loaderJobs/oldJobs, and that is much smaller. Should I copy the one from inWork to the parent directory? And now it shows -3498018815% so it is moving.
I'm thinking it may be because the disk filled up, which made me copy everything to a new disk. Does that sound logical?
Hmm... The actual download is from the file
tileproxy/loaderJobs/inWork/nais1.14, which is shortened by one line with each downloaded tile.
Common way:
The job file MapName.Zoom being copied form tileproxy/loaderJobs/ to tileproxy/loaderJobs/inWork/ by scheduler.
The scheduler runs the loaders, and the loaders shorten the file in tileproxy/loaderJobs/inWork/ to zero.
Then the scheduler deletes this file, make from file in the tileproxy/loaderJobs/ a file of the following zoom, deletes the previous and go again until the files in tileproxy/loaderJobs/ run out.Weird, after a reboot the file was there in loaderJobs, with a size of 40 bytes. That was the stuff I added to jumpstart the download. I copied over the file from the previous virtual disk's inWork, and then rebooted again. Now it shows a logical number (1 % done). So I think the problem started with the file missing in loaderJobs, which meant that it didn't start the job in loaderJobs/inWork. But adding some random tiles created a file that I first either overlooked or that wasn't shown, but I found after a reboot. Anyway, seems like the problem is fixed. Thanks! π
Something is happening here. This is in /var/log/apache2/error.log:
No big deal.
The Pi that's downloading C-Map has developed some very weird symptoms. I have set the max loader jobs in params.php to 5, but it's flooding the CPU with loader jobs and after a while it locks up, so I can't access the Pi from VNC anymore. At the same time it doesn't seem to download anything. The other Pi, wich is in the boat, isn't even able to download the C-Map tiles that I am watching in the regular way. Any idea what that can be? Can it be a change in C-Map's systems? I don't think it's a permission thing because both are mounted NTFS partitions, so they are always 0777.
You say about the C-MAP_raster1 map? I don't see any problems on this moment.
To check map availability run the
tilefromsource.php with --checkonly option. For example:
To automate this process, there is checkSources.php with brief doc inside. In Russian πThat doesn't look right to me...
Is it because I need to get a TOR setup for that as well now? It has been running without one.
It's ok. Just open "Get tile from:" bu right click.
Opened it in Chrome:
I get the same answer from another IP address (in my house) when running it there, so it's not my IP address that's blocked.
And also from a random NordVPN address.
I can ping the site:
In your launch the
tilefromsource.php above, a certain picture was successfully obtained. Just not the one listed in mapsources/C-MAP_raster1.php
On the file mapsources/C-MAP_raster1.php in the string "$trueTile=" replace 7e8245aa to 4ae8db71 and run
That seems to work, thanks. π But for some reason I still can't download the 17 zoom. Here I have parts of the map downloaded earlier, but it won't download futher tiles. And the other Pi, where I have the long job list of stuff to download from C-Map, is locking up withall those php calls I had in the earlier screenshot.
I think the one in the boatmay be a permission problem because no other maps work to download either (I can look at the maps that has been there from before, though), but I can 't figure out why, because when I look at the permissions it is root:root and "all" on both read, change and execute. That should really work, shouldn't it? Or is it because the www-data isn't technically a user?
You can check the availability of any tile:
OK, it is something with the NTSC permissions. I get it at least partially working when I create a new directory on the EXT4 partition and use that with the permissions set to anyone for modify. The weird thing is that it worked perfectly a few days ago.
Sorry, my mistake. Correctly:
So the tile present.
df -hi ?
ok
It's work for me.
It works when I do it with /GaladrielMap as a directory on the EXT partition too, but not when I do it with the SSD with the NTSC partition GaladrielMap mounted to the same place. So I know it's because of that. I just don't know why it has happened, because it worked fine to last week or maybe the week before. Is it possible that it can be because I have updated the Pi's?
Found it! The NTSC partitions on both the boat Pi and the download Pi had space left, but they were too fragmented, so they didn't accept data. I plugged the SSDs into a Windows computer and ran defragmentation there. Problem solved.
But I have another one that isn't that easy, I'm afraid. The virtual Debian that's downloading the NAIS-1 map refuses to download. It keeps updating the bannedSources file, and it contains the line of the last tile tried (I assume), like this: I have not changed anything in the TOR setup, but it does seem like it's not working. Is it possible that something has changed in either TOR setup or NAIS again? It seems like the TOR system is running, but I have no idea how to check if GaladrielMap is using it.
The mapsources/NAIS.php received some changes some time ago, download a fresh one.
Or edit existing. As I recall, it's need to specify the http version:
To make it work through the tor, you need to uncomment from
//'proxy'=>'tcp://127.0.0.1:8118' to 'proxy'=>'tcp://127.0.0.1:8118'
But it should work as it does.Thank you! π Banned sources is not updated, so I'm pretty sure it works now! Btw should I delete the 13xx.lock files and the .slock files? The are not updated either, they are there from before the change.
No, don't do it. It should work.
π
A totally different question: in the SignalK plugin version, how do I turn off all the options except for Follow to Cursor. So turn off these: ? The reason is that when I run GaladrielMap SK under Home Assistant it's unable to save the setup as a cookie, so I need the settings I use to be there every time I start it up. For some reason it seems to remember the zoom. I would like if possible to hide the button bar on the left side of the screen, but I don't know if that's even possible.
π¦
Oh, damn. I see now that my message was missing the important bits, sorry! All options in settings in the web interface, except for the "Follow to cursor", that one I want to keep. And if possible the whole buttton bar on the left side, so only the map and the zoom buttons are left. π
How do a map select without the left panel?
I won't. I will set it to NAIS1 and leave it at that. π
I have already set that as the default map in the plug-in settings in SignalK. So I just want a clean map.
It's quite time-consuming, but maybe...
I see. I was hoping it was as easy as commenting out a few things in the HTML.
OK, I killed the distance circles by removing the function:
What is the name of the left panel, is it possible to kill that in the same way? It seems like wind symbol and AIS won't bother me since the boat is offline when I use it.
I think I managed to remove wind the same way that I did the distance circles. π
Removing current track in the same way made the whole map black, so that didn't work, I think there are some if/then(else there that I don't understand, I'm no good at HTML. Just to make that clear, I'm not looking for a on/off solution, I just need to turn off the stuff I want and then I won't turn it back on.
Maybe it's easier to figure out why cookies aren't saved?
That I already know. Because they can't be in Home Assistant with Kiosk mode. But I got rid of the most annoying part, the distance circles.
This is how it looks on my 8" tablet, btw. The browser is WallPanel for Home Assistant, so everything you see is made in Home Assistant. Maps are fed by GaladrielCache, and they are running within KIP Instrument MFD, which gives me the necessary data at the bottom (speed, depth, engine RPM, and so on). At the moment of filming the boat and plotter weren't running, so only time and voltage are correct.
C-Map is a widget with the FreeboardSK plugin for KIP, while NAIS is GaladrielMap SignalK Edition in a webpage widget. I don't know why the change to Freeboard is so much slower, but it doesn't really matter because I only do the change to C-Map when I am ready to fish, and then back to NAIS when I'm done fishing. π
I have replaced the gps icon in GaladrielMap SignalK Edition with the boat from FreeboardSK and changed the size in the index.html so it's about the same. https://youtube.com/shorts/H35BLaeUXr8
Aha! I understood.
Maybe make the left panel disappear by tapping on the control, for example, in the right of the screen? You can to customise the display and then remove the panel.
That would be brilliant, thank you! π And since it doesn't save cookies it would be great if the same function would react to setting a parameter as well, either in the HTML or even in the settings fo the SignalK Plug-in.
BTW I think C-Map has run into a problem. It hasn't downloaded anything in a day or two, it's stuck on 63 % for raster 19, and I have "Forbidden" mesage in banned sources, which keeps updating:
I think C-Map has run into a problem.Heh. Yes, it is. It looks like Cloudflare has banned themselves π Use TOR, it works for me. Just set ExitNodes in USA or another democratic country.
Ah, I see! Thanks, will do!
Not so easy, sorry π¦
Cloudflare check the browser, appchart.c-map.com don't work too.
And that was not as easy as I thought... This is unfamiliar territory for me. I see that GaladrielMap starts TOR:
But I'm pretty sure it's not using it, because I still get the bans. I have these settings in the TOR section of my C-Map Source files and the torrc is the same as in the emergency kit. Is there something there or anywhere else I need to change?
Wait with that. Perhaps overcoming the Cloudflare will be enough.
Do you mean that you will try to change the mapsource file for C-Map?
Yes.
And Yes!!! Be patient π
I will try! π
About TOR:
First, install it as described.
Then add to torrc:
ExcludeExitNodes {ua}, {ru}, {no}
or
ExitNodes {gb}, {us}
Then add
ControlPort 9051
and
CookieAuthentication 0
Then add
HTTPTunnelPort 127.0.0.1:8118
Restart TOR.
If you want the tiles to be downloaded via the TOR you must uncomment the line
//'proxy'=>'tcp://127.0.0.1:8118',
to
'proxy'=>'tcp://127.0.0.1:8118',
in the map description file, _C-MAPraster1.php for example.It's kind of weird here with the file transfer...
It is the C-MAP_raster1.php file.
The only thing that was missing in my setup was to uncomment that line in C-Map_raster1.phpcd. But I still get that "No tile received" and banned sources keep updating.
Have you updated the C-MAP_raster1.php?
The updated version works for me.
Yes, I copied the "Spoiler" hidden text above and only changed the TOR part in it (uncommented proxy). Strange. I checked the torrc file again to make sure. This is the errror I get:
But there may be something strange about my TOR setup, this is the status, split in two because of the message lengt on Discord:
I can't understand why it says no authentication method, I thought this should be it: I tried with CookieAuthentication 1 as well. The error message disappeared, but I still get "No tile recived".
Oh, ifI use the TOR browser whatismyipaddress.com shows that I'm the middle of England.
No, it doesn't work more.
π¦
There is no easy way to overcome Cloudflare. In the appchart.c-map.com no maps for Russia, so I'm not really interested in fighting them.
I see. That's too bad. So TOR can not get around it?
Can I get around it by using NordVPN and change IP's every ten minutes? I don't have that much left of the C-Map I wanted to download. So how fast do an IP get banned?
No, it's different here. The Cloudflare is a tricky proxy. The website behind it is just like the website on your home computer. You can't just walk in from the outside.
The Cloudflare checks whether you walk using a browser. It is difficult to fully simulate the browser.
Aha, I see. π’ So no good way around it. And here I was 83 % done with my download, I should have started two days earlier on that one!
It's could make a browser plugin. π Maybe someday.
have you seen that cookies are now also required for c-map (cloudflare)?
Yes π All the usual methods have been tested.
The Cloudflare is a known problem. There's even a social movement against the Cloudflare. But the fact is, businesses don't like it when their website content isn't used the way they intended.
Maybe make the left panel disappear by tapping on the control, for example, in the right of the screen?See last version. Set up are in SignalK admin panel. This may not be what you wanted, but let's start with that.
Thank you, I will look at that later today or tomorrow! π Btw no news on the Cloudflare thing?
OK, I couldn't wait... π Tried it now, and it's very close to what I wanted, actually! If it could set it not to hide the zoom, it would be perfect for my use! Also where do I replace the boat icon for that? I like the boat looking icon for Freeboard, so I have that in the pure Galadriel Map, but I have not found the place to do that in the plug-in's files, as far as I can remember. Sorry if my memory fails me. Btw I could not set it to top right, but that is not a problem since I have the zoom there. So top middle works well. But the left panel is back next time I open the page or reload it, that is probably expected?
But one thing is strange: I have two Docker Containers of SignalK running to get C-Map and NAIS1 with a simple swipe of pages in KIP. And if I change the map in one of them (port 80) the other one (port 81) is changed too. Is that expected behaviour?
Btw no news on the Cloudflare thing?C-MAP_raster1 now works, but for me - only via TOR. Wait for the latest version - today or tomorrow. π I decided that disabling controls is a property of the client instance. So, the management of this has been moved to the GaladrielMap settings. To replace the boat icon replace
/usr/local/lib/node_modules/galadrielmap_sk/public/img/gpscursor.png
Yes, on window reload the controls are shown. Just to be safe.
No, it's something strange.
New 0.12.1. Store options to browser localStorage. So, you'll need to set it up again.
Sorry, 0.12.2Thank you! That works in a regular browser! But the map does not stick when I run it in a window in Home Assistant. I guess that's because it doesn't save cookies as they should in that case. And the boat icon is now as I like it. π
But the map does not stick when I run it in a window in Home Assistant. I guess that's because it doesn't save cookies as they should in that caseIts strange. In the new version should not use cookies. It should be used browser localStorage mechanism. Describe in detail what you are doing and what is happening.
Aha, that probably explains it. it's not a browser. It is Home Asistant with an embedded webpage. OK, in detail:
1. Home Assistant running on Wall Panel (kiosk browser available on GitHub and Play Store), with this YAML code:
2. KIP running as a webapp (to get the bottom line of info from plotter and outboard motor with Galadriel Map as an embedded webpage and this address: on the front page and this address when you move to the next page.
3. Two instances of SignalK running under Docker, both with Galadriel Map set up. With NAIS1 as the source in the first Docker instance (port 80) and C-Map in the second Docker instance (port 81).
The first pic in this post shows how the main screen on Wall Panel looks, and the second shows the first instance of SignalK/Galadriel map. The missing numbers at the bottom is because the plotter isnβt on, and the outboard isnβt running. The third pic is the empty map that I sometimes come to in Galadriel Map (I have not yet been able to find out what triggers it).
The problem is not with KIP or Home Assistant. The problem is with browser. One browser stores data from one url in one place. So all browser windows display the same things for this url.
To avoid this you must use Firefox's "containers". That is to open one instance of GaladrielMap (or KIP, or Freeboard) in one container, and other instance - in another container. It will not be possible to show two different GaladrielMap in one KIP.
But you may use different url's.
However, you already have different addresses...
Idon't think Firefox will make a difference, because nothing of this is actually rendered on the Pi. Home Assistant runs on Wall Panel kiosk browser, and the hardware is an Ouikitel rugged 8" tablet. This is what renders the web page. So it seems like Wall Panel does not save everything it should. And it is fully possible to show two different GaladrielMap in one KIP when I have one on port 80 and one on port 81. So yes, they have different addresses. π But I think it's pretty safe to say that the culprit is Wall Panel browser, so you don't have to spend more time on that thing. The only thing I would like, if possible, is that the I can set the sidebar not to show on reload of the Galadriel Map web page.
I'm sure we should leave the ability to use the app to someone who doesn't know how to use it. So no, after reload you must hide controls again.
OK, is there a way around it? Like a flag that I can set in a config file?
Sorry, I think I've missed the point of the problem.
After reload, you just can tap to hide controls again. The location of the tap zone is saved after reload.
Yeah, I can. I just wanted to get rid of the tapping altogether. But OK, if it's not practictal, no biggie.
So I'm thinking - what if there's an emergency and the boat has to be driven by someone who doesn't know how things work? It's better if he sees the controls after some commonly understood actions.
That is of course a totally different use from mine. I use the NAIS1 map when I drive the 30 minutes out to where I check my crawfish pots, use C-Map there (because the bottom contours are better on that) and then go back to NAIS1 when I want to go back. Sometimes the tablet looses the connection to the network for a few seconds, and that means the map is reloaded. So the controls are back.
Sometimes the tablet looses the connection to the network for a few seconds, and that means the map is reloaded.Aha! But the GaladrielMap not need to be reload, and does not do it when the connection lost. Is it possible that the KIP does this?
Actually I think it's the kiosk browser that does it. ANd I need to have this inside of Home Assistant because I control just about everything else from there (heavy metal over the speakers, lights, lanterns, pumps and so on is all run with Home Assistant and ESP32's with tons or relays - I have to be close to 50 relays in the boat!).
Ok, the latest version now restores the hidden status after reload.
Wow, thank you! π But the update refuses to install. Logs:
Well, it's not me. I guess.
Weird thing. I have tried several times, and I have updated SignalK docker container too. Could the full log help? This is what it says:
Remove the GalagrielMap, and then install it again.
Or just remove
/root/.signalk/node_modules/galadrielmap_sk.Thanks, thatlast version worked, and then just installing with npm. π
OK, weird. A reboot, and wants to upgrade again, says it is 12.2, not 12.3. I deleted the version on the host and ten copied the version NPM installed in the Docker image from the image to the host. Where can I find the version number for the plug-in, so I know it's the right one? In readme.md it says 0.11. Oh, btw, I still couldn't update from the plug-in store in SignalK, same error.
The current version is shown in the header of the browser window from the file
/usr/local/lib/node_modules/galadrielmap_sk/package.json.
The SugnalK store gets app from https://www.npmjs.com/package/galadrielmap_sk
And I forgot to change the version in the doc.
Remove the /root/.signalk/plugin-config-data/galadrielmap_sk.json.
I did not find any other problems when upgrading on the test machine.No, it still fails. Do you have a test machine running SignalK in Docker, preferably with Docker Compose? I suspect that may be the problem. Anyway, I finally managed to get it working on one of my two instances, but the other one still gives me trouble. Trying to remove it from the Appstore throws an error as well:
I have no idea.
I have two test machines: one real and one virtual. No problems with any of them.
But it is not Docker, yes.
Then I think the problem is somewhere in the docker system.