Kevin Powell - Community•7d ago

Creating an IPFS UnixFS File System By Hand

I am contemplating a collaborative file system based on IPFS. Users will publish the root of a directory tree & a service worker will copy that tree into a Neo4j instance running in Docker with a model of a versioned repository in it. The frontend, then, will talk to Neo4j (which is a graph database) to allow more complex views of the data than a simple tree view. I'm particularly interested in incorporating ordered lists where users rank content. Say I could get 50 people's opinions of the top 20 funniest memes ever, & generate a composite list from those. To avoid having to upload a ton of information for every update, I want to reuse the unchanged parts of the previous iteration that are already in IPFS rather than reupload. To do this, I need to manually construct the directory tree. I had it working with a CBOR-DAG, but the ipfs dag commands are only available on instances where you have API access. So, I want to switch to UnixFS to store stuff. I've tried to get Claude, V0, Co-Pilot, & Bolt to help me, and this is where we're at currently:

#!/usr/bin/env node

import { create as ipfsClient } from 'kubo-rpc-client'
import { prepare, encode } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'

const ipfs = ipfsClient('http://localhost:5001')

try {
  const content = '¡Hello, World!'
  const file = new UnixFS({
    type: 'file',
    data: (
      typeof content === 'string' ? (new TextEncoder().encode(content)) : (content)
    ),
  })
  const fileNode = prepare({ Data: file.marshal(), Links: [] })
  const fileBytes = encode(fileNode)
  const start = await ipfs.add(fileBytes, { layout: 'dag-pb' })
  console.info(`Start CID: ${start.cid.toString()}`)

  const dir = new UnixFS({ type: 'directory' })
  const dirNode = (
    prepare({
      Data: dir.marshal(),
      Links: [{ Hash: start.cid, Name: 'hello.txt', Tsize: file.fileSize() }],
    })
  )
  const dirBytes = encode(dirNode)
  const final  = await ipfs.add(dirBytes, { layout: 'dag-pb' })
  console.info(`Final CID: ${final.cid.toString()}`)
} catch(error) {
  console.error(`Error: ${error.message}`)
  console.error(error.stack)
}

#!/usr/bin/env node

import { create as ipfsClient } from 'kubo-rpc-client'
import { prepare, encode } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'

const ipfs = ipfsClient('http://localhost:5001')

try {
  const content = '¡Hello, World!'
  const file = new UnixFS({
    type: 'file',
    data: (
      typeof content === 'string' ? (new TextEncoder().encode(content)) : (content)
    ),
  })
  const fileNode = prepare({ Data: file.marshal(), Links: [] })
  const fileBytes = encode(fileNode)
  const start = await ipfs.add(fileBytes, { layout: 'dag-pb' })
  console.info(`Start CID: ${start.cid.toString()}`)

  const dir = new UnixFS({ type: 'directory' })
  const dirNode = (
    prepare({
      Data: dir.marshal(),
      Links: [{ Hash: start.cid, Name: 'hello.txt', Tsize: file.fileSize() }],
    })
  )
  const dirBytes = encode(dirNode)
  const final  = await ipfs.add(dirBytes, { layout: 'dag-pb' })
  console.info(`Final CID: ${final.cid.toString()}`)
} catch(error) {
  console.error(`Error: ${error.message}`)
  console.error(error.stack)
}

The output I get from that is:

Start CID: Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
Final CID: QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
❯ ipfs dag get Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
{"Data":{"/":{"bytes":"CAISFwoVCAISD8KhSGVsbG8sIFdvcmxkIRgPGBc"}},"Links":[]}
❯ ipfs dag get QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
{"Data":{"/":{"bytes":"CAISNRIvCiISIMvU/WQImfeyfmFxszzo6JKoCnjTRTSoY4rD1esrAmr+EgloZWxsby50eHQKAggBGDU"}},"Links":[]}
❯ ipfs cat Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm | cat -A
$
^U^H^B^R^OM-BM-!Hello, World!^X^O
❯ ipfs cat QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx | cat -A
^R/$
"^R M-KM-TM-}d^HM-^YM-wM-2~aqM-3<M-hM-hM-^RM-($
xM-SE4M-(cM-^JM-CM-UM-k+^BjM-~^R^Ihello.txt$
^B^H^A

Start CID: Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
Final CID: QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
❯ ipfs dag get Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
{"Data":{"/":{"bytes":"CAISFwoVCAISD8KhSGVsbG8sIFdvcmxkIRgPGBc"}},"Links":[]}
❯ ipfs dag get QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
{"Data":{"/":{"bytes":"CAISNRIvCiISIMvU/WQImfeyfmFxszzo6JKoCnjTRTSoY4rD1esrAmr+EgloZWxsby50eHQKAggBGDU"}},"Links":[]}
❯ ipfs cat Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm | cat -A
$
^U^H^B^R^OM-BM-!Hello, World!^X^O
❯ ipfs cat QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx | cat -A
^R/$
"^R M-KM-TM-}d^HM-^YM-wM-2~aqM-3<M-hM-hM-^RM-($
xM-SE4M-(cM-^JM-CM-UM-k+^BjM-~^R^Ihello.txt$
^B^H^A

So, my file attempt has a bunch of random crap in with the data, & my directory doesn't have any children. When I open either up in the IPFS Desktop it says they're not browsable.

0 Replies

No replies yetBe the first to reply to this messageJoin

Gaming

Programming

Creating an IPFS UnixFS File System By Hand