Topics

Cloudflare Developers•2y ago

My logpush just stopped pushing logs It

My logpush just stopped pushing logs.. It's an http source and it says pushing

No description

37 Replies

DanTheGoodmanOP•2y ago

It has worked historically, but logs are just not being pushed. I just tried recreating it. I can see the test event comes through when creating the job but I am not getting any requests to it when creaming logs. This is what my creation command looks like:

curl -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNTID/logpush/jobs" \
-H "X-Auth-Key: $APIKEY" \
-H "X-Auth-Email: $EMAIL" \
-H "Content-Type: application/json" \
-d "{
  \"name\": \"workers-logpush\",
  \"output_options\": {
    \"field_names\": [\"ClientIP\", \"EdgeStartTimestamp\", \"RayID\", \"Event\", \"EventTimestampMs\", \"Outcome\", \"Exceptions\", \"Logs\", \"ScriptName\"],
    \"sample_rate\": 1.0,
    \"timestamp_format\": \"unixnano\",
    \"output_type\": \"ndjson\",
    \"batch_prefix\": \"[\",
    \"batch_suffix\": \"]\",
    \"record_prefix\": \"{\",
    \"record_suffix\": \"}\",
    \"record_delimiter\": \",\"
  },
  \"filter\": \"{\\\"where\\\":{\\\"key\\\":\\\"ScriptName\\\",\\\"operator\\\":\\\"\!eq\\\",\\\"value\\\":\\\"cloudflareaxioslogpush\\\"}}\",
  \"destination_conf\": \"https://$SUBDOMAIN.workers.dev/log?header_Authorization=Bearer%20$TOKEN&header_content-type=application%2Fjson\",
  \"dataset\": \"workers_trace_events\",
  \"enabled\": true
}"

curl -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNTID/logpush/jobs" \
-H "X-Auth-Key: $APIKEY" \
-H "X-Auth-Email: $EMAIL" \
-H "Content-Type: application/json" \
-d "{
  \"name\": \"workers-logpush\",
  \"output_options\": {
    \"field_names\": [\"ClientIP\", \"EdgeStartTimestamp\", \"RayID\", \"Event\", \"EventTimestampMs\", \"Outcome\", \"Exceptions\", \"Logs\", \"ScriptName\"],
    \"sample_rate\": 1.0,
    \"timestamp_format\": \"unixnano\",
    \"output_type\": \"ndjson\",
    \"batch_prefix\": \"[\",
    \"batch_suffix\": \"]\",
    \"record_prefix\": \"{\",
    \"record_suffix\": \"}\",
    \"record_delimiter\": \",\"
  },
  \"filter\": \"{\\\"where\\\":{\\\"key\\\":\\\"ScriptName\\\",\\\"operator\\\":\\\"\!eq\\\",\\\"value\\\":\\\"cloudflareaxioslogpush\\\"}}\",
  \"destination_conf\": \"https://$SUBDOMAIN.workers.dev/log?header_Authorization=Bearer%20$TOKEN&header_content-type=application%2Fjson\",
  \"dataset\": \"workers_trace_events\",
  \"enabled\": true
}"

And I've waited more than long enough for it to batch the logs, they are just not shipping right now

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

Correct, let me get job id @kagitac 184337 I've been socketed for 5 hours to the worker and no logs have come through, which means no logs have been pushed to it

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

2.9.1 and no it does not, but logs did push previously

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

deploying now! Do i need to recreate the logpush job @kagitac ?

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

AH! there they come, so the fact that it worked before was a bug maybe lol

DanTheGoodmanOP•2y ago

@kagitac while I have you here, are you familiar with this? Happens when ever I make a new http destination. It makes a test body but says it's JSON in the content type

No description

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

today, actually lol So my worker looks like this:

export interface Env {
    TOKEN: string
    DATASET: string
}

export default {
    async fetch(
        request: Request,
        env: Env,
        ctx: ExecutionContext
    ): Promise<Response> {
        if (request.headers.get("content-length") === "4") {
            console.log("got test", env.DATASET, env.TOKEN)
            return new Response("xxx")
        }
        if (request.headers.get("content-encoding") === "gzip") {
            // This is a logpush batch
            console.log("got logs")

            // Verify token
            if (request.headers.get("Authorization")?.split("earer ")[1] !== env.TOKEN) {
                return new Response("invalid token", {
                    status: 401
                })
            }

            return fetch(`https://cloud.axiom.co/api/v1/datasets/${env.DATASET}/ingest`, {
                method: "POST",
                headers: {
                    "Authorization": `Bearer ${env.TOKEN}`,
                    "content-encoding": "gzip",
                    "content-type": "application/json"
                },
                body: request.body
            })
        }
        return new Response("ok")
    },
}

export interface Env {
    TOKEN: string
    DATASET: string
}

export default {
    async fetch(
        request: Request,
        env: Env,
        ctx: ExecutionContext
    ): Promise<Response> {
        if (request.headers.get("content-length") === "4") {
            console.log("got test", env.DATASET, env.TOKEN)
            return new Response("xxx")
        }
        if (request.headers.get("content-encoding") === "gzip") {
            // This is a logpush batch
            console.log("got logs")

            // Verify token
            if (request.headers.get("Authorization")?.split("earer ")[1] !== env.TOKEN) {
                return new Response("invalid token", {
                    status: 401
                })
            }

            return fetch(`https://cloud.axiom.co/api/v1/datasets/${env.DATASET}/ingest`, {
                method: "POST",
                headers: {
                    "Authorization": `Bearer ${env.TOKEN}`,
                    "content-encoding": "gzip",
                    "content-type": "application/json"
                },
                body: request.body
            })
        }
        return new Response("ok")
    },
}

I originally made it send right to axiom but that expects JSON, that probably sent a 400 my 401s are probably me needing to update the token sec recreated job with. id 184454

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

@kagitac it seems like the auth headers are not sending the auth header I am getting is null, just logigng everything now to check I will reset the token later yeah ok something about my vars was scuffed, just fixing the job 184458 @kagitac no thats a check in case the Bearer part is not caps correctly lol. The whole auth header is just Bearer right now not sure why because my token injection is working... to the logpush creation command looks like it's retrying a bunch of old jobs? I'm def getting spammed witha ton lol pushing a fix to temp swallow them @kagitac working now

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

yeah for a while it seems lol

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

totally fair lol, and luck on the test request? I'd love to just send directly to axiom instead of the worker

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

no worries, sounds good! also question, why limit to 150 chars per log line? @kagitac ? It's a bit painful if I need to jump a body for example, or have more than 2 uuids

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

is it planning to be increased then?

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

awesome. Do logs through the r2/s3 integrations come in full?

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

Yeah I almost made a logging package that ships logs off to GCP before ending the worker but they require fs and such in their package lol XD didn't really want to add the extra latency

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

I just worry about the extra latency, esp since that's over the internet :/ I'd love to use queues to try it but that is limited to 100req/s and I can def have bursts over that

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

yeah blocking the response from the worker while I am shipping logs

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

I can run it async? OH MY GOD have to try that, then I can make a generic logger where I can just implement a "ship" function. It will log to console in real time then call that and will batch ship to a destination tysm for being so helpful!!!

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

so there you process the index in the background?

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

DanTheGoodmanOP•2y ago

Oh wow lol that’s awesome

Unknown User•2y ago

Message Not Public

Sign In & Join Server To View

Cloudflare Developers Join

Welcome to the official Cloudflare Developers server. Here you can ask for help and stay updated with the latest news

60KMembers

View on Discord

Want results from more Discord servers?

Add your server