Telodendria raises a bad pointer when logging in with Nheko #33

Closed
opened 2023-09-09 10:52:18 +00:00 by lda · 8 comments
Contributor

Currently, when trying to login using Nheko to a Telodendria server running on GNU/Linux(while spoofing the supported versions, since Nheko doesn't like anything >v1.5), Telodendria will hit a bad pointer:

HttpRouter.c:291: Bad pointer to 0 bytes of memory at [ptr].,

which happens to be when Telodendria is freeing all of the matches for the routing part(in HttpRouterRoute), with the last route handled in the logs being GET /_matrix/client/v3/room_keys/version

Trying to see the stacktrace using GDB also seems to show that this happens when trying to match the path \a(memory corruption going on?).

Currently, when trying to login using Nheko to a Telodendria server running on GNU/Linux(while spoofing the supported versions, since Nheko doesn't like anything >v1.5), Telodendria will hit a bad pointer: `HttpRouter.c:291: Bad pointer to 0 bytes of memory at [ptr].`, which happens to be when Telodendria is freeing all of the matches for the routing part(in `HttpRouterRoute`), with the last route handled in the logs being `GET /_matrix/client/v3/room_keys/version` Trying to see the stacktrace using GDB also seems to show that this happens when trying to match the path `\a`(memory corruption going on?).
Author
Contributor

Extra information about everything's here.

GDB bt output of the core dump:

(gdb) bt
#0  0x0000000000443fcc in __pthread_kill_implementation.constprop.0 ()
#1  0x0000000000428a48 in raise ()
#2  0x0000000000405b1e in TelodendriaMemoryHook (a=MEMORY_BAD_POINTER, i=0x7f6d00013870, args=0x0)
    at Telodendria.c:114
#3  0x0000000000422d49 in MemoryFree (p=0x7f6d0000d1a4, file=0x4fdff0 "HttpRouter.c", line=291) at Memory.c:346
#4  0x000000000041979b in HttpRouterRoute (router=0x7f6d10003d34, path=0x7f6d00002884 "\a", args=0x7f6d07fff170, 
    ret=0x7f6d07fff148) at HttpRouter.c:291
#5  0x00000000004039f0 in MatrixHttpHandler (context=0x7f6d00001d24, argp=0x7f6d17029130) at Matrix.c:81
#6  0x000000000041fcba in HttpServerWorkerThread (args=0x7f6d0c000ff4) at HttpServer.c:578
#7  0x0000000000442a06 in start_thread ()
#8  0x000000000047ed9c in clone3 ()

Telodendria log file(level set to debug):

[23-09-09 13:45:45 (0) *] Configuration:
[23-09-09 13:45:45 (0) *]   Server Name: localhost:8008
[23-09-09 13:45:45 (0) *]   Base URL: http://localhost:8008
[23-09-09 13:45:45 (0) *]   Identity Server: (null)
[23-09-09 13:45:45 (0) *]   Run As: (null):(null)
[23-09-09 13:45:45 (0) *]   Max Cache: 0
[23-09-09 13:45:45 (0) *]   Flags: 9
[23-09-09 13:45:45 (0) *] HTTP listener: 0
[23-09-09 13:45:45 (0) *]   Port: 8008
[23-09-09 13:45:45 (0) *]   Threads: 4
[23-09-09 13:45:45 (0) *]   Max Connections: 32
[23-09-09 13:45:45 (0) *]   Flags: 0
[23-09-09 13:45:45 (0) *]   TLS Cert: (null)
[23-09-09 13:45:45 (0) *]   TLS Key: (null)
[23-09-09 13:45:45 (0) *] Running as uid:gid: 1000:1000.
[23-09-09 13:45:45 (0) *] No user/group info specified in the config.
[23-09-09 13:45:45 (0) !] Database caching is disabled.
[23-09-09 13:45:45 (0) !] If this is not what you intended, check the config file
[23-09-09 13:45:45 (0) !] and ensure that maxCache is a valid number of bytes.
[23-09-09 13:45:45 (0) *] Registering jobs...
[23-09-09 13:45:45 (0) ~] Starting job scheduler...
[23-09-09 13:45:45 (0) ~] Building routing tree...
[23-09-09 13:45:45 (1) *] User Interactive Auth sessions: 0
[23-09-09 13:45:45 (0) ~] Starting server...
[23-09-09 13:45:45 (0) *] Started HTTP server 0.
[23-09-09 13:45:45 (0) >] Listening on port: 8008
[23-09-09 13:45:45 (0) *] Installed signal handler: SIGINT
[23-09-09 13:45:45 (0) *] Installed signal handler: SIGTERM
[23-09-09 13:45:45 (0) *] Installed signal handler: SIGPIPE
[23-09-09 13:45:45 (0) *] Installed signal handler: SIGUSR1
[23-09-09 13:45:56 (2) *] GET /_matrix/client/v3/profile/%40lda%3Alocalhost%3A8008
[23-09-09 13:45:56 (3) *] GET /_matrix/client/v3/room_keys/version
[23-09-09 13:45:56 (3) >] GET /_matrix/client/v3/room_keys/version (404 Not Found)
[23-09-09 13:45:56 (2) x] HttpRouter.c:291: Bad pointer to 0 bytes of memory at 0x7f6d0000d1a8.

Weirdly enough, I could only replicate this with Nheko. Trying my way to send requests myself never caused Telodendria to crash.

Extra information about everything's here. GDB `bt` output of the core dump: ``` (gdb) bt #0 0x0000000000443fcc in __pthread_kill_implementation.constprop.0 () #1 0x0000000000428a48 in raise () #2 0x0000000000405b1e in TelodendriaMemoryHook (a=MEMORY_BAD_POINTER, i=0x7f6d00013870, args=0x0) at Telodendria.c:114 #3 0x0000000000422d49 in MemoryFree (p=0x7f6d0000d1a4, file=0x4fdff0 "HttpRouter.c", line=291) at Memory.c:346 #4 0x000000000041979b in HttpRouterRoute (router=0x7f6d10003d34, path=0x7f6d00002884 "\a", args=0x7f6d07fff170, ret=0x7f6d07fff148) at HttpRouter.c:291 #5 0x00000000004039f0 in MatrixHttpHandler (context=0x7f6d00001d24, argp=0x7f6d17029130) at Matrix.c:81 #6 0x000000000041fcba in HttpServerWorkerThread (args=0x7f6d0c000ff4) at HttpServer.c:578 #7 0x0000000000442a06 in start_thread () #8 0x000000000047ed9c in clone3 () ``` Telodendria log file(level set to `debug`): ``` [23-09-09 13:45:45 (0) *] Configuration: [23-09-09 13:45:45 (0) *] Server Name: localhost:8008 [23-09-09 13:45:45 (0) *] Base URL: http://localhost:8008 [23-09-09 13:45:45 (0) *] Identity Server: (null) [23-09-09 13:45:45 (0) *] Run As: (null):(null) [23-09-09 13:45:45 (0) *] Max Cache: 0 [23-09-09 13:45:45 (0) *] Flags: 9 [23-09-09 13:45:45 (0) *] HTTP listener: 0 [23-09-09 13:45:45 (0) *] Port: 8008 [23-09-09 13:45:45 (0) *] Threads: 4 [23-09-09 13:45:45 (0) *] Max Connections: 32 [23-09-09 13:45:45 (0) *] Flags: 0 [23-09-09 13:45:45 (0) *] TLS Cert: (null) [23-09-09 13:45:45 (0) *] TLS Key: (null) [23-09-09 13:45:45 (0) *] Running as uid:gid: 1000:1000. [23-09-09 13:45:45 (0) *] No user/group info specified in the config. [23-09-09 13:45:45 (0) !] Database caching is disabled. [23-09-09 13:45:45 (0) !] If this is not what you intended, check the config file [23-09-09 13:45:45 (0) !] and ensure that maxCache is a valid number of bytes. [23-09-09 13:45:45 (0) *] Registering jobs... [23-09-09 13:45:45 (0) ~] Starting job scheduler... [23-09-09 13:45:45 (0) ~] Building routing tree... [23-09-09 13:45:45 (1) *] User Interactive Auth sessions: 0 [23-09-09 13:45:45 (0) ~] Starting server... [23-09-09 13:45:45 (0) *] Started HTTP server 0. [23-09-09 13:45:45 (0) >] Listening on port: 8008 [23-09-09 13:45:45 (0) *] Installed signal handler: SIGINT [23-09-09 13:45:45 (0) *] Installed signal handler: SIGTERM [23-09-09 13:45:45 (0) *] Installed signal handler: SIGPIPE [23-09-09 13:45:45 (0) *] Installed signal handler: SIGUSR1 [23-09-09 13:45:56 (2) *] GET /_matrix/client/v3/profile/%40lda%3Alocalhost%3A8008 [23-09-09 13:45:56 (3) *] GET /_matrix/client/v3/room_keys/version [23-09-09 13:45:56 (3) >] GET /_matrix/client/v3/room_keys/version (404 Not Found) [23-09-09 13:45:56 (2) x] HttpRouter.c:291: Bad pointer to 0 bytes of memory at 0x7f6d0000d1a8. ``` Weirdly enough, I could *only* replicate this with Nheko. Trying my way to send requests myself never caused Telodendria to crash.
Owner

Thanks for the logs.

I wonder if this is some sort of concurrency issue somewhere, because it looks like thread 2 throws the error after thread 3 has returned a 404.

Is this pretty reliably replicated? That is, does it do this every single time, or does it behave differently sometimes?

Thanks for the logs. I wonder if this is some sort of concurrency issue somewhere, because it looks like thread 2 throws the error after thread 3 has returned a 404. Is this pretty reliably replicated? That is, does it do this every single time, or does it behave differently sometimes?
jordan added the
bug
question
labels 2023-09-09 11:53:04 +00:00
Owner

I wonder if this is related to #21. Obviously not a duplicate, because this is a heck of a lot more detailed than #21 is, but that'd be nice if fixing this and any other specific problems we have make #21 no longer an issue.

I wonder if this is related to #21. Obviously not a duplicate, because this is a heck of a lot more detailed than #21 is, but that'd be nice if fixing this and any other specific problems we have make #21 no longer an issue.
Author
Contributor

Is this pretty reliably replicated? That is, does it do this every single time, or does it behave differently sometimes?

I've tried a few times, and it seems like it happens 100% of the time with Nheko.

> Is this pretty reliably replicated? That is, does it do this every single time, or does it behave differently sometimes? I've tried a few times, and it seems like it happens 100% of the time with Nheko.
Owner

Great! Issues like this are so much easier to debug. If you want to try to debug this one, go ahead, and feel free to ask questions. Otherwise, I'll plan on tackling this sometime after I finish the Gitea Migration, or at least get far enough into it that I feel comfortable with Telodendria's online presence.

Great! Issues like this are so much easier to debug. If you want to try to debug this one, go ahead, and feel free to ask questions. Otherwise, I'll plan on tackling this sometime after I finish the [Gitea Migration](https://git.telodendria.io/Telodendria/telodendria/projects/3), or at least get far enough into it that I feel comfortable with Telodendria's online presence.
Author
Contributor

Noting that down for posterity: Logging in every memory action Telodendria does(by removing the part after the && in the "prologue" of TelodendriaMemoryHook seems to make the issue go away, though this looks like quite the red herring to me(and also is a pretty bodge-y solution)

Also: I doubt it is a threading issue, as it still seems to crash when setting the thread count and max connections to 1.

EDIT: It seems like that from previous testing, you don't even have to log memory actions, and it only fixes itself when the condition: (a == MEMORY_ALLOCATE) || (a == MEMORY_REALLOCATE) || (a == MEMORY_FREE) is removed, although I haven't tested this as much.

I think I am going to investigate this a bit more tomorrow with other tools than GDB.

Noting that down for posterity: Logging in *every* memory action Telodendria does(by removing the part after the `&&` in the "prologue" of `TelodendriaMemoryHook` seems to make the issue go away, though this looks like quite the red herring to me(and also is a pretty bodge-y solution) Also: I doubt it is a threading issue, as it still seems to crash when setting the thread count and max connections to 1. EDIT: It seems like that from previous testing, you don't even have to *log* memory actions, and it only fixes itself when the condition: `(a == MEMORY_ALLOCATE) || (a == MEMORY_REALLOCATE) || (a == MEMORY_FREE)` is removed, although I haven't tested this as much. I think I am going to investigate this a bit more tomorrow with other tools than GDB.
Owner

Good thinking with setting threads and connections to 1. Not sure why changing the logging configuration would make the issue go away, that's really strange.

Good thinking with setting threads and connections to 1. Not sure why changing the logging configuration would make the issue go away, that's really strange.
Author
Contributor

After a bit of debugging a few hours ago, I think I was able to determine that matches added with this code(in HttpRouterRoute):
ArrayAdd(matches, StrSubstr(pathPart, pmatch[i].rm_so, pmatch[i].rm_eo)); somehow become invalid before the freeing part, thus causing a bad pointer issue.
I have still to determine why exactly is it becoming invalid.

Edit: Just found the culprit, making a PR soon.

After a bit of debugging a few hours ago, I *think* I was able to determine that matches added with this code(in `HttpRouterRoute`): `ArrayAdd(matches, StrSubstr(pathPart, pmatch[i].rm_so, pmatch[i].rm_eo));` *somehow* become invalid before the freeing part, thus causing a bad pointer issue. I have still to determine *why* exactly is it becoming invalid. Edit: Just found the culprit, making a PR soon.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Telodendria/Telodendria#33
No description provided.