Is it possible to get maps in an easier way than MapBox?
I tried to make a small map of my area at the tip of Norway, from Kristiansand to Lyngdal (which is as far as I can imagine going even when I retire in a few years) on MapBox, but I gotta admit that I gave up. The learning curve is steep. I wanted to use TukTuk in SignalK to have an extra, fully offline map available on the phone/tablet in addition to the one on the Garmin 720 in the (open) wheelhouse, to know exactly where the depth changes are when setting crawfish and crab pots. But is there a simpler way to get what I need than MapBox?
182 Replies
you could try opencpn & download mbtiles with sasplanet. On android only though. Sasplanet has navionics and satellite images which it can save in a format opencpn can read. Sat images are great!! π
Thanks, but I'm not looking for something to work only one device. I need it on the SignalK server, so it's accessable on all my devices, which includes a Windows Surface tablet/laptop.
You can create mbtiles using sasplanet and use the mbtiles using signalk/freeboard possibly will work with tuktuk as well. It is a nice solution especially if you interested in a map for a small region
You should not use the SAS.Planet...
Three steps:
1) Install GaladrielCache on your signalk computer
2) Configure charts-plugin to use GaladrielCache
3) Explore your region on the map via TukTuk (or freeboard-sk or GaladrielMapSK)
After some time you will have the required map for off-line use.
@Ba2 The tiles work for Tuktuk as well, yeah. The reason I want to use that is that I liked the way it showed overlays fit me better.
@ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ Thanks! I'll have a look at that. I will do a full image backup of the Pi first, and then install it. Maybe I can find the cahce too and save that on my SSD.
Having looked at it now I'm afraid that is a bit over my level, I don't understand how to install and use it.
Thanks! I need to find out how to install php and Apache, I have never used anything like that.
apt install apache2 php π
But need PHP7, not PHP8.Thanks! I'll try!
I think I have set it up according to the instructions (except for the TOR connection, which I don't have, I don't think I need that since I won't be using AIS), but I get "The requested URL not found", It's probably me being thick, I have tried with http://127.0.0.0/galadriel-map, http://127.0.0.0/galadrielmap and http://127.0.0.0/map but none of these work.
If you are on the same computer, then:
http://127.0.0.1/map/
It is a GaladrielMap chartplotter.Fantastic, thank you very much! It works! Now the final question, where do I find the cached map tiles? I looked around, but didn't see anything. I have not made a special card for it, so there is nothing as sdb1 at the moment. Do I need to use that, or will the map tiles end up another place?
/GaladrielMap/tileproxy/tiles/
but if you want to configure Signal K Charts plugin for use this cache, you must create 'Online chart provider' and set URL to http://127.0.0.1/tileproxy/tiles.php?z={z}&x={x}&y={y}&r=OpenTopoMap for OpenTopo, for example.
Then all features of the GaladrielCache will be available from any SignalK chartplotter.Great, thanks again!
I see that loading the maps is very slow, especially the NAIS Norwegian that I would like to use. But I'll just let it go until it's down.
This may be a mass download ban. I haven't checked. To overcome - you need TOR π
BTW, for Navionics you must get a fresh
/GaladrielMap/tileproxy/mapsources/ files from fresh zip.Thanks, but I don't need Navionics, I am good with the Norwegian maps, those are the ones I'm used to. And I do have a NordVPN account. Maybe that will help? π€
I think I'm going to set up a virtual Debian on my server and run it from there to see if that helps. π
It didn't, so now I have installed Tor. I just need to find out how I connect to it. This is something I've never used before.
And using the file that came with Galadriel-map, it says "Tor quit during startup". I have added the data lines from the torrc file that was created by the system at the top of the file in the zip, but nothing. Sorry if I'm stupid, but I just can't get it working.
Also how can I see in GaladrielMap that it's going via Tor? Or do I do that by using the Tor browser to connect to Galadriel Map?
You should just replace the original torrc with a torrc from the zip.
The torrc from zip should work, but it's better to comment out the lines "UseBridges" and "Bridge" at the bottom of file if the TOR is not blocked in your country.
The GaladrielMap suite uses the TOR for two purposes:
1) As transport for netAIS
2) As proxy for GaladrielCache. The idea is to change the "exit node" every time after downloading several tiles.
The bad news is that this mechanism is disabled everywhere. To enable it, you need to edit the map source description file.
The second bad news is that such a mechanism is not implemented for NAIS Norwegian map.
I can add the required to map source description file for NAIS Norwegian map tomorrow.
Done. See new https://github.com/VladimirKalachikhin/Galadriel-map/blob/master/emergencykit/GaladrielMap-2.9_GaladrielCache-2_gpsdPROXY-0_netAIS-1_NoVectorTiles_.zip
I improved the map source description file for NAIS Norwegian map. It works fine for me without TOR proxy. But the ability to use a TOR proxy has been added too.
Thank you so very much for your patience and help! Now it floats like crazy here! I am afraid I will go all completist and try to cache the whole coast of Norway! π€£
@ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ One question that I stubmled over today: When i move this over to the boat, from the virtual Debian I'm using to get the map tiles, I would like to move the whole GaladrielMap directory to my SSD, to make it independent of the OS on the micro SD. I edited the file apache2_galadrielmap.conf and replaced all instances of with but I get (I use port 8080 to allow SignalK to keep port 80, but that is not a problem, at least not on the virtual Debian). And I have edited it one place in params.php. Is there another file where I need to edit the path?
I see that the error message refers to the original directory: So there has to be somewhere I haven't changed it yet. I have changed the directory statements in the apache2_galadrielmap.conf file, but left the alias, at least I think that's correct. Like this:
Thanks! I tried that version first, using the full directory on all the aliases, but that didn't work either.
Same error message, "Forbidden". I have checked the rights to the directory /media/pi/Docker/GaladrielMap and even changed it to allow anybody to change it, but to no avail.
Server configuration is magic.
The rights to the directory are not enough, you need the rights to the files:
chmod -R ...Yeah, and I'm afraid I'm no magician. I followed the chown and chmod in the instructions again, this time with the -R swithc, like this:
Still the same "Forbidden"'. π’ But now I think I'm closer, in the Apache log I see: I'm sure that tells you something, even if I don't understand it.
I tried and that shows that most are while a few are Shouldn't they all be the same after the chmod?
I also tried after a tip on search, but nothing. That made it worse.
You set
chown -hR :www-data /media/pi/Docker/GaladrielMap, but not a chown -hR :www-data /media.
User www-data has no permission to /media/pi/Docker. But this path part is present in the Directory option.Aha, I see! I think the setup har a bit too many directories to keep track of. I think I will try to use the original directory, but try to mount a separate partition on the SSD to that directory. Thanks!
You can set a "ServerRoot" directory, and use relative paths.
Ah, thanks! That sounds better!
Hm... Still not working, and I have done the chown again recursively for GaladrielMap. I have set the directories to , and I have changed both document root in sites-available/000-default.conf and Directory in apache2.conf But I still get the Forbidden. I tried too, but with the same result. Sorry if I'm dense, I just haven't done this at all before...
"ServerRoot" is server option, not directory. It is located in the global
apache2.conf. But it's not fashionable to use it now.
It is possible not to configure "Alias", just symlink GaladrielMap/map/ dir as map/ to default Apache www dir (/var/www/?, set in apache2.conf)
The www path is not a filesystem path, so common way:
1) give the permissions to the filesystem path
2) describe www path as "Alias" or set www path === file system pathA separate partition for" GaladrielMap" mounted to root turned out to be the smoothest solution. Nothing to change except for mounting it in fstab. Thanks again for your hjelp! π
It was invented for a router where there is no "disk", but there is a SD slot. But this is a simple and universal solution.
I have one problem left, it seems. I have GPSD on a separate pi, with the address 192.168.3.2:2947. Is it possible to add that somewhere? From what I can see in the params.php there's only an option for local GPSD.
/GaladrielMap/map/gpsdPROXY/params.php
$dataSourceHost
$dataSourcePort
Just the local gpsd is a default.Thank you! I changed that, but I can't see my boat. Is there something else I shoould change? I have put in this:
The "//" is comment. You must:
My bad, I keep expecting the # comment from Python or -- from LUA... But I'm afraid it's still not working. Is there something else I should try? And is there a error log somewhere that I can look at and maybe find out why it doesn't connect? I know the addresses and ports are correct, I am using GPSD both in SignalK, Homeassistant and Node-RED.
Heh, you set $dataSourceHost twice, and the second time - on SignalK which the GNSS receiver is not connected to, as far as I understand.
Of course I had... So now I have removed that, but still no fun. Should I edit anything in ?
No.
Is your gpsd on 192.168.3.2:2947? Check it by
telnet 192.168.3.2 2947. Must be {"class":"VERSION","release":"3.22","rev":"3.22","proto_major":3,"proto_minor":14}
Has the gpsdPROXY been restarted after changing the config? Try pkill -f gpsdPROXY.php and reload GaladrielMap window in browser. Do ps -A | grep gpsd. Is gpsdPROXY run?Theres https://map.openseamap.org/ but they are very map not chart
Works with leaflet if youre just trying to visualise things rather than nav
@ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ I get the correct answer on Telnet from GPSd. The whole Pi was restarted. The pkill did not change anything. But gives nothing, so I think it's not running. @JonR I'm afraid that's useless for me. I need precise depth curves down to at least 200 meters for setting my crawfish pots (I'm a very eager hobby fisher), and as far as I know only Norwegian charts gives me that.
Ok,
You can see some info or error.
So it says it's running, but there is nothing on grep.
π It's ok. π see
sudo crontab -u www-data -l
The gpsdPROXY cannot connect to the gpsd. Check the /GaladrielMap/map/gpsdPROXY/params.phpShould the crontab command give a result? It gave nothing. And I have attached the full contents of the file here. π
It should work...
Maybe I'm missing something. So, you open http://localhost/map/? And you see the map, but you don't see your position?
Actually I'm opening http://192.168.3.1:8080/map/ (the address of the boat's Pi, and 8080 because I have SignalK on port 80).
And I see the map, but no position.
(Oh, for the moment I'm opening it from another network segment, I have the 3.x segment routed via my Server, the main segment is 2.x.)
But http://localhost:8080/map/ doesn't give me the position either. Is it because I'm using the 8080 port for the map, so that has to be changed somewhere else too?
No.
Open http://192.168.3.1:8080/map/ in the browser, and open the browser console (Ctrl-Shift-i). See about "spatialWebSocket".
Does the
telnet 192.168.3.2 2947 from the 192.168.3.1 connect to gpsd?Yes, I telneted from 3.1 to 3.2 in the test above. Same result now:
Heh. http is tricky.
Replace the line
if($gpsdProxyHost=='localhost' or $gpsdProxyHost=='127.0.0.1' or $gpsdProxyHost=='0.0.0.0') $gpsdProxyHost = $_SERVER['HTTP_HOST'];
with the line
if($gpsdProxyHost=='localhost' or $gpsdProxyHost=='127.0.0.1' or $gpsdProxyHost=='0.0.0.0') $gpsdProxyHost = $_SERVER['SERVER_NAME'];
in the /GaladrielMap/map/index.html file.
This is line number 1155
Oh, sorry: in /GaladrielMap/map/index.php file.Bingo! You have a winner! π π π That works, thank you very much! Btw can I have NAIS as the default map instead of OpenTopo? I always start up in the middle of Moscow with OpenTopo when I open. π Even though it now moves to my actual position within a second or two, I still have OpenTopo.
π¦ Must be a last map(s) and a last position.
I see. OK, no biggie, the important stuff is fixed now! Thank you again!
The last map and the last position is saved in the browser cookies. Nothing special. Maybe you don't allow cookies?
That is very possible, I have strict privacy enforced. I'll make sure to whitelist the map!
These are my fishing grounds, and the chart is correct almost on the meter:
The picture above is I just happened to find another Norwegian map.
Hi again, @ΠΠ»Π°Π΄ΠΈΠΌΠΈΡ ΠΠ°Π»Π°ΡΠΈΡ
ΠΈΠ½ ! Suddenly I can't download any more of the NAIS map. Can you please check if it works for you? I have tried both with and without Tor, and it doesn't change anything.
Heh. HTTP/1.1 500 Internal server error.
In the official (https://status.geonorge.no/cache.html) is same.
But the Russian hackers know the way. Be patient.
By the way
Checking tile availability:
php tilefromsource.php -z15 -x17360 -y9532 -rNAIS --maxTry=15 --checkonly
Checking all maps availability:
php checkSources.php
, see checkSources.log, use cron.
Prepare map source file to checkSources.php:
$trueTile=...
Read the docks, they are rules.
As far as I understand, the cache service of geonorge.no is broken. Forever or for a while, I do not know.
I can give you the same map from the original(?) server, but it's slower and the tiles are 1.5 times larger. Or wait for a while, suddenly it will work.Thank you! That was annoying, maybe something in an update messed it up? I will wait a few weeks, and I'll get back to this thread if it's not in by then.
Oh, I went to the website, and it says that there is a new version of UML model (whatever that means) because they have excluded "GrunnGass/pockmarks" (whatever that is). That probably means that they won't be back. Can the original server stuff be mixed with what I already have downloaded, or do I need to download everything again for that?
Yes, it looks like it.
Here are two other sources for this map. It seems to be the same visually, but files is different. Both have larger files.
In the NAIS1 the map is called "sjokartraster2", in the kartverket.no_sjokartraster - "sjokartraster". Perhaps NAIS1 is a new map, and kartverket.no_sjokartraster is a copy of the old.
I think you need to make sure which map is visually "the same", and then rename the corresponding file to NAIS.php
Or create symlink from
tiles/NAIS1/ or tiles/kartverket.no_sjokartraster/ to tiles/NAIS/Thank you very much, at least NAIS1 works, and it seems like it can work with themap I've already downloaded ! π Its not slow either. But one question: Is it possible to make Galadriel map download and cache the whole NAIS1 map in a way? Sort of pretend I'm looking at all the whole map in all zoom levels? Perhaps with the tilefromsource script in some way?
it seems like it can work with themap I've already downloadedNo. If you have not taken any action, then
tiles/NAIS and tiles/NAIS1 is different directory.
Is it possible to make Galadriel map download and cache the whole NAIS1 map in a way?"Download" menu item.
I copied all the files from NAIS to NAIS1, and they seem to load as they should even if NAIS1 has another format. And with the Download I thought I had to scroll over the map in the correct zoom levels. Or have I not found the correct numbers to add to the empty box?
I copied all the files from NAIS to NAIS1, and they seem to load as they should even if NAIS1 has another format.It's the way... But raster tiles is the raster tiles: it just 255x255 png.
And with the Download I thought I had to scroll over the map in the correct zoom levels. Or have I not found the correct numbers to add to the empty box?Dowloader load tiles from the specified zoom to max zoom (16 by default). The common use is to specify the tiles covering the required area at a suitable viewing zoom (9, for example)(easy by tap on number). Then all tiles from this zoom to the max zoom will be downloaded.
Got it, thanks! π π
Something weird is stopping me from downloading, maybe it's just me being stupid again. But I don't seem to get any maps in nais1 or even C-MAP when I try to download by selecting parts of the map, only when zooming in. Is there some kind of download queue that I may have blocked with something, that I should try to clean?
Does nothing change at all? Is that stays 0%?
Yeah, on download. The map parts only actually downloads when I zoom in on them.
Ok, try on server
cd /GaladrielMap/tileproxy
php loader.php
and look what's going on.It is ok.
What it shows
ls /GaladrielMap/tileproxy/loaderJobs
ls /GaladrielMap/tileproxy/loaderJobs/inWork
?Something is strange...
cd /GaladrielMap/tileproxy
php loaderSched.php
?
ps w -A | grep php ?Replace file
/GaladrielMap/tileproxy/loader.php with the file from the archive and try php loader.php.Do
chmod 775 /GaladrielMap/tileproxy/loaderJobsDone. And restarted Apache 2. This is the result: So it's half the size now.
ps w -A | grep loaderSched ?ls /GaladrielMap/tileproxy/loaderJobs
ls /GaladrielMap/tileproxy/loaderJobs/inWork
?We need to see what the loaderSched does, but there is a problem: the loaderSched can't just be killed, it is restarted by cron.
So you have to kill the loaderSched in one terminal and quickly start it in another terminal.
In the 1-st terminal type:
sudo kill 1497
In the 2-st terminal
cd /GaladrielMap/tileproxy
and type
php loaderSched.php
After it press Enter in 1-st terminal, and, after killing, quickly press Enter in 2-st.Do
chmod -R 775 /GaladrielMap/tileproxy/loaderJobs
πGreat, it is making progress, thanks! π I first got fooled by the percentages to think that it had stopped, but I see now that I have to close the download tab and reopen it for the numbers to update.
There is no need to keep updating percentage - it's not really important information.
However, as I recall, you seem to have set the correct permissions on those directories?
I did follow the instructions all along, so something has changed the permissions later, but I have no idea what that can be.
By the way: Do you have any desire to make a Norwegian localization of the program?
I could try. Is it many lines of text?
I hope I don't have to look through the code to find the lines to translate from English to Norwegian, that will be a disaster since I am no good at this coding.
About 170 lines π
It;s easy:
cp /GaladrielMap/map/internationalisation/en.php /GaladrielMap/map/internationalisation/no.php
and translate all or some phrases to Norwegian.I will have a look at it. π
OK, looked at it. π I think that should be doable, yeah. Just to make sure, in lines with an = sign I translate what's within single quotes after that sign, right? And in lines without that = I also translate what's within the single quotes?
Yes, all phrases in single quotes.
If your browser is not forced to use English - you will see the result immediately after reload page.
OK, I will look at it one of the first days. π
Weird... I did a few translations as a test, but I still only get the English wording on the download page. I have not have a set language, and it is the same from several devices with different browsers.
On the GaladrielMap page: right click -> View Page Source
On the opened "view-source" page: line 2 must be '<html lang="no">'
Is that true?
No, the line is "en" for some reason, on both Chrome, Firefox and Vivaldi. And that goes for both the "download" installation that I have on a virtual machine on my server and the installation on the Pi in the boat.
Ok, on a GaladrielMap page press Ctrl-Shift I, select Console and then reload the page.
Find in Console (somewhere at the top) line like this:
GET http://192.168.10.10/map/
Open dropdown.
Find "Request headers" and show the line "Accept-Language"
I don't see any of that on reload. Only error messages:
Instead the "Default levels" select "Verbose" on dropdown.
Well, the error is clear to me.
You will have to install a fresh version later.
Heh, I already fixed it? Which version are you using?
Sorry, I have had some problems, but I'm back again now. I will install the newest version this evening and see if that fixes it. Should I replace everything (and fear another permissions problem), or is there a specific file I can replace?
It's easier to update everything with the
unzip GaladrielMap-2.10_GaladrielCache-2.9_gpsdPROXY-0_netAIS-1.zip "GaladrielMap/*" -d /
But! Keep the params.php files if you have changed them.I had to set up the virtual Debian again, something went wrong with the partition. I had it working again, but then I installed the lastest version and tried to open the map, I was stuck at a zoom of 3000 and the zoom buttons were greyed out, no matter what map I tried.
Which map are you looking at?
Are there the same problems with another map?
Yeah, seems to be all maps. This is CMap:
Without any map loaded the zoom buttons work, but if I zoom in to 500 and then load a map it jumps right back to 3000 and greyed out buttons.
Turn off ALL maps.
Wait 20 sec.
Reload page.
Turn any map on.
But actually, I couldn't reproduce it.
Nope, that didn't change anything. Weird. I will copy back the setup I had there before and try to upgrade once again.
Ok.
Turn off ALL maps.
Wait 20 sec.
Reload page.
Turn on the "OpenTopoMap".
Tried it again now, and I did exactly what you say over, but same result. 3000 km zoom and greyed out buttons. Can it be related to having the partition for Galadriel Map with the NTFS filesystem? I discovered that ext4 on a VMWare virtual machine got filled up with the huge number of small PNG files.
(I have a separate partition for GaladrielMap, mounted as /GaladrielMap btw, just like I have had all the time.) Another thing that can be different is that I have not installed Tor yet on that newly restored VM. I will do that now.
Show me what's in the browser console (Ctrl-Shift-I) when the page loads?
OK, the problem was not having Tor installed. It works now, with Tor installed and running. I'll leave it running until tomorrow afternoon to see if it downloads map tiles om nais1 as it should. I'll get beack to you then.
OK, tried now, I still do not see it. This is with verbose and all other levels activated:
Scratch that, I found it in Firefox. This is what you're looking for, I think:
I tried to change the filename to nb-NO.php, but that didn't help.
Just
nb.phpThere we go! I now have my translated words. I'll do most of that tomorrow, I think. π
Btw there's some Russian in the English file. Like here, what do those words mean?
This is for the international crew. Or local rescuers... I'm not sure it's necessary at all.
Open the
ru.php and look at the corresponding variable. There's an English text.Are you still here? I heard that Russia has blocked Discord. But with your knowledge of TOR that's probably not a problem. π I have translated the file, I will read proof on the translation tomorrow. Wing in ground confused me, I had never heard about such a thing! Turns out that it's called "ekranoplan" in Norwegian, like it is in Russian, I believe. And I kept the extra text on course and position in English, as it is in the Russian file, that seemed most logical to me.
Via TOR it works, but TOR is blocked too. π The world has long been mad...
Yes, "Wing in ground" is strange, and yes, it is "ekranoplan" in Russian.
I'm going nuts here... Suddenly the latest version does not work at all! I have GaladrielMap on a separate virtual disk, mounted as /GaladrielMap. When I try the virtual disk that I had before the last version, it works. But when I try with the virtual disk with the latest version I either get "internal server error 500" or a blank page. And in that previous version the translation does not show. But it's not the translation, it doesen't work without the nb.php file either.
So I don't change anything in the rest of the system, I just replace one virtual disk with another. Can you please tell me where to look for the problem? Is it possible that it's an update for Tor that has broken it? There has been a couple the last days.
/var/log/apache2/error.log ?PHP Parse error: syntax error, unexpected '','(T_CONSTANT_ENCAPSED_STRING), expecting ')' in /GaladrielMap/map/internationalisation/nb.php on line 95
Extra quotation in the line 94 of the nb.php?You're right, of course! FIxed. I will have the Norwegian file ready in less than an hour. π
I think I must wait until tomorrow, I need to check if there is an official version of the AIS codes.
Got that, now the last detail: "Loader runs" and "loader not runs" (I'd suggest changing that to "loader running" and "loader not running" in Eglish) , is that the same as the downloader or something else?
Btw I don't hink I've said it, but I work as a translator and have been for 35 years, so it should be pretty correct. π
Oh, another thing: What file do I change to make the default map nais1 instead of Opentopo, and the default position to my position? When I open on a new device (and I have lots of devices...) I always start up on Opentopo in the middle of Mosocw.
Got that, now the last detail: "Loader runs" and "loader not runs" (I'd suggest changing that to "loader running" and "loader not running" in Eglish) , is that the same as the downloader or something else?Yes, it's Downloader status, but... I found that this variable is not used anywhere else. I have to think about whether it's right. And yes, my English is very bad π¦
Oh, another thing: What file do I change to make the default map nais1 instead of Opentopo, and the default position to my position?You have already asked: you need to allow cookies in browsers.
Your English is far from bad, actually. π I know I asked about a similar question, this is actually different. I want to have the nais1 as the default when I open the map on a new device that hasn't opened the map before.
I want to have the nais1 as the default when I open the map on a new device that hasn't opened the map before.Ah! I understood. I didn't think anyone would need it. It's hardcoded in the index.php in
var defaultMap = 'OpenTopoMap';
string (# 575); and
if(! startCenter) startCenter = L.latLng([55.754,37.62]);
string (# 583).
But it's probably better to to have it in the params.php I will do this in the next version.
Here it isIt works π
Great, I'm glad it works! I have my VM downloading everything on the NAIS1 map from zoom 6 to 16. It's using 80-90 % of the CPU on the VM! π And thanks, I'll wait until you have another version out, then. It's not something that I'm in a hurry to get.
I have my VM downloading everything on the NAIS1 map from zoom 6 to 16. It's using 80-90 % of the CPU on the VM!
/GaladrielMap/tileproxy/params.php :
$maxLoaderRuns = 15; // simultaneously working loader tasks.
Set it to 10 for less using CPU.Hi again! I have another question: How do I stop the loader jobs and restart them later? Stopping apache2 does not stop them, and disabling apache2 on startup does not block them from startup on a reboot. I would like to control this, but I can't find out how. π
This is what I mcgyverd together and it works great.
Followed https://digidem.notion.site/Preparing-and-Adding-Custom-Basemaps-to-Mapeo-b4f13019f0b842ce9315c6097f08ce36#b125e5b6233045448bac5fd28197f03e
1. Debian Bookwork (12)
2. install base and deps
apt install gdal-bin htop vim iotop npm python3-pip python3-full
npm install -g tl
npm fund
npm install -g @mapbox/mbtiles tilelive-file tilelive-http @mapbox/tilelive @mapbox/tilejson
npm fund
npm install -g tl tilelive-http tilelive-file
npm fund
more /usr/share/doc/python3.11/README.venv
mkdir -p ~/.venvs
python3 -m venv ~/.venvs/mbutil
~/.venvs/mbutil/bin/python -m pip install mbutil
3. Generate Charts
sample
tl copy -z 12 -Z 18 -b "-75.6 -1.6 -74.05 -0.275" http://ecn.t0.tiles.virtualearth.net/tiles/a{q}.jpeg?g=5162 file://./siekopai-peru-bing-tiles?filetype=jpg
tl copy -z 8 -Z 12 -b "133.896887 6.818803 134.764801 7.8" http://ecn.t0.tiles.virtualearth.net/tiles/a{q}.jpeg?g=5162 file://./palau-bing-tiles?filetype=jpg
tl copy -z 15 -Z 15 -b "133.896887 6.818803 134.764801 7.8" http://ecn.t0.tiles.virtualearth.net/tiles/a{q}.jpeg?g=5162 file://./palau-bing-tiles?filetype=jpg
- Where the -z and -Z refer to the lowest and highest zoom levels downloaded,
- The coordinates of the bounding box (w, s, e, n, in WGS 1984 unprojected).
- The file://./ is followed by the file name you want it to have
cd palau-bing-tiles
rm unwanted zoom levels in the subdir
mv metadata.json metadata.orig
/root/.venvs/mbutil/bin/mb-util --image_format=jpg /root/palau-bing-tiles/ /root/palau.mbtiles
gdalinfo palau.mbtiles
--> Test in OpenCPN
NOTES:
Bruce usually uses z8 till Z18 without Z17 (he uses: Z8 Z10 Z12 Z15 Z18)
hHanks, but I have a very good solution for the Norwegian NAIS map, and I have around 80 gig of that downloaded, so I think I'll stick to GaladrielMap. I just need to know how to pause the download.
How do I stop the loader jobs and restart them later?Tap to green "lamp" on the Loader tab to stop, and to red - to start.
Aha, thanks! I never thought about using the GUI/webpage!
Heh, sorry, NO!
Its do not pause, its completely stops!
Generally speaking, there is no way to suspend it. Everything is done there so that it downloads anyway.
You can
kill loaderSched.php and all loader.php, and after start loaderSched.php for resume.
Sorry again: It is impossible to kill loaderSched.php. Need first remove it from cron.
But if you create job file and start download from GaladrielMap - the copy of the job file saved in tileproxy/loaderJobs/oldJobs/.
You can start the download again - only missing tiles will be downloaded.I figured I could just comment out the cron job and made www-data an account that could log in. But as that user the command shows an empty file. I have set the correct permissions for the directory /var/www and a password that the account accepted, and I'm able to choose nano as the editor, it says that "No modification made", but no error messages. Any idea why? Is the cron jobitself running from another user, and running the php stuff as the www-data user?
Do you really need to pause the download? If you turn off the computer and then turn it on, the download will continue. If the download is really loading your computer - reduce the number of download threads by edit $maxLoaderRuns in
tileproxy/params.php
Another way to pause the download is remove and backup job files from tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/ Then the loader will kill itself.
To resume loading return job files to tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/ and start loaderSched.php from GaladrielMap or by tileproxy/startLoaderDaemon or just php loaderSched.phpThe problem is that even with only 4 threads it is so heavy that it kills my Pi's connection from the boat to the cabin when downloading CMap. But I guess I can move this download as well to something inside the cabin, another Pi or virtual Debian . The virtual Debian is busy with the NAIS map, which goes around 1 % every two days, up to 12 now. I know it's silly to want the full map of Norway, but I plan to do a bit more long distance traveling in a few years, when I retire.
Something strange... There is no difference which map to download, except maybe Transas.
Which specific map is causing the problem?
It's not the difference in the map, it's that I was downloading CMap directly on the Pi in the boat (while NAI1S was downloaded on a virtual Debian on the server). But yesterdayafternoon I moved that to a pi in the house with LAN instead of WiFi, and that seems to have solved it.
The problem is that even with only 4 threads it is so heavy that it kills my Pi's connection from the boat to the cabin when downloading CMap.If you mean C-MAP_raster1.php, it has a bug. Use the attached one.
I see! Thank you! π
I'm going crazy... I had to move the whole GaladrielMap directory to another virtual disk because the one I had was full, and something weird has happened with the permissions or something else. I did run afterwards and but it doesn't start downloading again. In the map's download page it's stuck on "download started", and the bullet is grey, not green or red. Trying to run the commands that you told me (killing the process and starting it again) I get this error, which is a bit different from the one I had earlier with the wrong permission:
Can you please tell me what's happening there? I checked the owner and permissions on the original VM, and it seems like they are the same..
php loaderSched.php from what user? Is he www-data or in the www-data group?
And so - the scheduler started, it started the loaders....I don't know, really. But it was done from the regular user, so I guess not. I used the stuff earlier up from you:
We need to see what the loaderSched does, but there is a problem: the loaderSched can't just be killed, it is restarted by cron. So you have to kill the loaderSched in one terminal and quickly start it in another terminal. In the 1-st terminal type:sudo kill 1497In the 2-st terminalcd /GaladrielMap/tileproxyand typephp loaderSched.phpAfter it press Enter in 1-st terminal, and, after killing, quickly press Enter in 2-st.
Heh, there are some bugs there, but I don't think it's about them π
See to
tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/.
If you see tileproxy/loaderJobs/chartName.Zoom and do not see correspondent tileproxy/loaderJobs/inWork/chartName.Zoom file - you have a problem.
Maybe you need to set write permissions to tileproxy/loaderJobs/ and tileproxy/loaderJobs/inWork/ for everyone.I have this, and the directories and files have write permissions for everyone. I have been running which took more than 24 hours with all the map tiles in there. Can it be something about all those lock files?
The .lock files is PID of loaders. Currently running or, may be, dead.
.slock is a PID of scheduler.
The scheduler will delete the outdated .lock files.
The grey download indicator means that no are loader jobs and the loader is not running.
Weird. The nais1.14 file in contains 1,3 GiB with stuff to download.
I selected some stuff I had already downloaded and clicked on the "Add to download" button, and it turned green. And it is still green. Maybe that kickstarted in in a way? The download percentage is a bit weird, though... π€ But I'm pretty sure it has started, the CPU usage has gone up to where it was while downloading. I'' leave it over night and see what it says on the percentage tomorrow.
By "Add to download" you start download from www-data user.
The negative percentage mean that
tileproxy/loaderJobs/inWork/nais1.14 is bigger than tileproxy/loaderJobs/nais1.14. That is, these are different files.As I tried to say with the screen shot, which shows all files in the directory loaderJobs there is no file in loaderJobs, only in loaderJobs/inWork. The only file with the same name is in /loaderJobs/oldJobs, and that is much smaller. Should I copy the one from inWork to the parent directory? And now it shows -3498018815% so it is moving.
I'm thinking it may be because the disk filled up, which made me copy everything to a new disk. Does that sound logical?
Hmm... The actual download is from the file
tileproxy/loaderJobs/inWork/nais1.14, which is shortened by one line with each downloaded tile.
Common way:
The job file MapName.Zoom being copied form tileproxy/loaderJobs/ to tileproxy/loaderJobs/inWork/ by scheduler.
The scheduler runs the loaders, and the loaders shorten the file in tileproxy/loaderJobs/inWork/ to zero.
Then the scheduler deletes this file, make from file in the tileproxy/loaderJobs/ a file of the following zoom, deletes the previous and go again until the files in tileproxy/loaderJobs/ run out.Weird, after a reboot the file was there in loaderJobs, with a size of 40 bytes. That was the stuff I added to jumpstart the download. I copied over the file from the previous virtual disk's inWork, and then rebooted again. Now it shows a logical number (1 % done). So I think the problem started with the file missing in loaderJobs, which meant that it didn't start the job in loaderJobs/inWork. But adding some random tiles created a file that I first either overlooked or that wasn't shown, but I found after a reboot. Anyway, seems like the problem is fixed. Thanks! π
Something is happening here. This is in /var/log/apache2/error.log:
No big deal.
The Pi that's downloading C-Map has developed some very weird symptoms. I have set the max loader jobs in params.php to 5, but it's flooding the CPU with loader jobs and after a while it locks up, so I can't access the Pi from VNC anymore. At the same time it doesn't seem to download anything. The other Pi, wich is in the boat, isn't even able to download the C-Map tiles that I am watching in the regular way. Any idea what that can be? Can it be a change in C-Map's systems? I don't think it's a permission thing because both are mounted NTFS partitions, so they are always 0777.
You say about the C-MAP_raster1 map? I don't see any problems on this moment.
To check map availability run the
tilefromsource.php with --checkonly option. For example:
To automate this process, there is checkSources.php with brief doc inside. In Russian πThat doesn't look right to me...
Is it because I need to get a TOR setup for that as well now? It has been running without one.
It's ok. Just open "Get tile from:" bu right click.
Opened it in Chrome:
I get the same answer from another IP address (in my house) when running it there, so it's not my IP address that's blocked.
And also from a random NordVPN address.
I can ping the site:
In your launch the
tilefromsource.php above, a certain picture was successfully obtained. Just not the one listed in mapsources/C-MAP_raster1.php
On the file mapsources/C-MAP_raster1.php in the string "$trueTile=" replace 7e8245aa to 4ae8db71 and run
That seems to work, thanks. π But for some reason I still can't download the 17 zoom. Here I have parts of the map downloaded earlier, but it won't download futher tiles. And the other Pi, where I have the long job list of stuff to download from C-Map, is locking up withall those php calls I had in the earlier screenshot.
I think the one in the boatmay be a permission problem because no other maps work to download either (I can look at the maps that has been there from before, though), but I can 't figure out why, because when I look at the permissions it is root:root and "all" on both read, change and execute. That should really work, shouldn't it? Or is it because the www-data isn't technically a user?
You can check the availability of any tile:
OK, it is something with the NTSC permissions. I get it at least partially working when I create a new directory on the EXT4 partition and use that with the permissions set to anyone for modify. The weird thing is that it worked perfectly a few days ago.
Sorry, my mistake. Correctly:
So the tile present.
df -hi ?
ok
It's work for me.
It works when I do it with /GaladrielMap as a directory on the EXT partition too, but not when I do it with the SSD with the NTSC partition GaladrielMap mounted to the same place. So I know it's because of that. I just don't know why it has happened, because it worked fine to last week or maybe the week before. Is it possible that it can be because I have updated the Pi's?
Found it! The NTSC partitions on both the boat Pi and the download Pi had space left, but they were too fragmented, so they didn't accept data. I plugged the SSDs into a Windows computer and ran defragmentation there. Problem solved.
But I have another one that isn't that easy, I'm afraid. The virtual Debian that's downloading the NAIS-1 map refuses to download. It keeps updating the bannedSources file, and it contains the line of the last tile tried (I assume), like this: I have not changed anything in the TOR setup, but it does seem like it's not working. Is it possible that something has changed in either TOR setup or NAIS again? It seems like the TOR system is running, but I have no idea how to check if GaladrielMap is using it.
The mapsources/NAIS.php received some changes some time ago, download a fresh one.
Or edit existing. As I recall, it's need to specify the http version:
To make it work through the tor, you need to uncomment from
//'proxy'=>'tcp://127.0.0.1:8118' to 'proxy'=>'tcp://127.0.0.1:8118'
But it should work as it does.Thank you! π Banned sources is not updated, so I'm pretty sure it works now! Btw should I delete the 13xx.lock files and the .slock files? The are not updated either, they are there from before the change.
No, don't do it. It should work.
π