Authentication is not working from fluentd-->Loki-->Grafana

[Manoharan-NMS]

Configuration File Details

Fluentd
File: /etc/td-agent/td-agent.conf

<match huge.log>
  @type loki
  format json
  url "http://localhost:3100"
  username test
  password test

  <buffer>
    @type memory
    chunk_limit_size 8m
    compress gzip
  </buffer>

  flush_at_shutdown true
  flush_mode interval
  flush_interval 5s
  flush_thread_count 10

  extra_labels {"job":"huge.log", "host":"workstation_hl_new", "agent":"fluentd_hl_new" }
  <label>
      filename
  </label>
</match>

Loki
File: loki-local-config.yaml

auth_enabled: true

Grafana:

Under loki datasource Basic Auth Details given

Expected:
Able to communicate through Authentication.

Actual Error in Grafana

logger=context userId=1 orgId=1 uname=admin t=2022-11-29T14:56:15.50091764+05:30 level=error msg="Failed to call resource" error="no org id\n" traceID=
logger=context userId=1 orgId=1 uname=admin t=2022-11-29T14:56:15.501022502+05:30 level=error msg="Request Completed" method=GET path=/api/datasources/5/resources/labels status=500 remote_addr=[::1] time_ms=1 duration=1.690432ms size=51 referer=http://localhost:3000/datasources/edit/EWm6GTdVz handler=/api/datasources/:id/resources/*
``

[Manoharan-NMS]

Error Observed in td-agent.log

2022-11-29 14:49:45 +0530 [warn]: #0 failed to flush the buffer. retry_times=9 next_retry_time=2022-11-29 14:58:23 +0530 chunk="5ee985c214f8709c188392fece03ee99" error_class=Errno::ECONNRESET error="Connection reset by peer"
  2022-11-29 14:49:45 +0530 [warn]: #0 suppressed same stacktrace

[eeddaann]

hey, the provided logs in td-agent.log seems related to this issue which related to the buffering and not the authentication.

can you try to use a small logfile instead the huge one for troubleshooting?
if the small file will work, tweaking the buffering configuration might solve your issue.

waiting for your update :)

[Manoharan-NMS]

hey, the provided logs in td-agent.log seems related to this issue which related to the buffering and not the authentication.

can you try to use a small logfile instead the huge one for troubleshooting? if the small file will work, tweaking the buffering configuration might solve your issue.

waiting for your update :)

td-agent.conf

<source>
  @type tail
  format json
  path "/data/test_log_new/new.txt"
  pos_file /var/log/td-agent/app_test.pos
  tag app_test
  #read_from_head true
</source>

<filter app_test>
  @type record_transformer
  enable_ruby
  <record>
    avg ${record["total"] / record["count"]}
    application_type ${tag}
  </record>
</filter>

<filter _test.huge.log>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    tag ${tag}
    newvalue "TEST_APP"
  </record>
</filter>

<match app_test>
  @type loki
  format json
  url "http://localhost:3100"
  username test
  password test

  <buffer>
    @type memory
    chunk_limit_size 8m
    compress gzip
  </buffer>

  flush_at_shutdown true
  flush_mode interval
  flush_interval 5s
  flush_thread_count 10

  extra_labels {"job":"New_Parser", "host":"apptest.com", "agent":"fluentd" }
  <label>
      filename
  </label>
</match>

/var/log/grafana/grafana.log

logger=context userId=1 orgId=1 uname=admin t=2022-11-30T13:42:56.675542181+05:30 level=error msg="Failed to call resource" error="no org id\n" traceID=
logger=context userId=1 orgId=1 uname=admin t=2022-11-30T13:42:56.675652739+05:30 level=error msg="Request Completed" method=GET path=/api/datasources/5/resources/labels status=500 remote_addr=[::1] time_ms=1 duration=1.69881ms size=51 referer=http://localhost:3000/datasources/edit/EWm6GTdVz handler=/api/datasources/:id/resources/*

[Manoharan-NMS]

@eeddaann Still its not working . Currently parsed one file which has few lines only.