unicode-org icu - Super huge icudt72l_dat!

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
Post Reply
User avatar
AndrewAPrice
Member
Member
Posts: 2300
Joined: Mon Jun 05, 2006 11:00 pm
Location: USA (and Australia)

unicode-org icu - Super huge icudt72l_dat!

Post by AndrewAPrice »

Has anyone else dealt with porting https://github.com/unicode-org/icu? The stuff inside of icu4c/source/data builds a super huge icudt72l_dat.S, that when statically linked into my applications, causes the binaries to grow by 43MB! :evil:

It's read inside of source/common/udata.c in doLoadFromCommonData and openCommonData via the macro U_ICUDATA_ENTRY_POINT. I noticed the macro ICU_DATA_DIR_WINDOWS, but I'm going to have to dig into it a little more to see if I can move it to a file. And perhaps implement processes sharing the memory of files.
My OS is Perception.
nexos
Member
Member
Posts: 1081
Joined: Tue Feb 18, 2020 3:29 pm
Libera.chat IRC: nexos

Re: unicode-org icu - Super huge icudt72l_dat!

Post by nexos »

Does your OS support shared libs? If so, then ICU can be a shared lib, and that will reduce the amount of memory / disk space used.
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg
User avatar
AndrewAPrice
Member
Member
Posts: 2300
Joined: Mon Jun 05, 2006 11:00 pm
Location: USA (and Australia)

Re: unicode-org icu - Super huge icudt72l_dat!

Post by AndrewAPrice »

Unfortunately I don't have support for shared libraries. For now, I'd like to load it from a file. I'll change the U_ICUDATA_ENTRY_POINT macro to be a pointer to a function that lazily loads the data file into memory, then returns a pointer to it.

(My microkernel does support shared memory between processes. For large read-only data files, perhaps I could make a service (maybe the VFS) that reads files into shared memory. I'll have to update my shared memory to support one-writer/multiple-readers.)
My OS is Perception.
User avatar
AndrewAPrice
Member
Member
Posts: 2300
Joined: Mon Jun 05, 2006 11:00 pm
Location: USA (and Australia)

Re: unicode-org icu - Super huge icudt72l_dat!

Post by AndrewAPrice »

My OS is Perception.
User avatar
AndrewAPrice
Member
Member
Posts: 2300
Joined: Mon Jun 05, 2006 11:00 pm
Location: USA (and Australia)

Re: unicode-org icu - Super huge icudt72l_dat!

Post by AndrewAPrice »

I got it to build with the archive file. (I use a custom build system so I'm unravelling what the make rules do and reimplementing it in my system. I had to link in stubdata but then set the macro to the directory where I put the generated .dat file.)

I ported icu because it's a dependency of Skia. I actually have Skia building and drawing my UI widgets now, except text. I have an issue with fontconfig not being able to parse my fonts.conf.
My OS is Perception.
User avatar
AndrewAPrice
Member
Member
Posts: 2300
Joined: Mon Jun 05, 2006 11:00 pm
Location: USA (and Australia)

Re: unicode-org icu - Super huge icudt72l_dat!

Post by AndrewAPrice »

I've linked in the stubdata and provided the .dat archive on disk. I'm logging file accesses in my vfs server and noticed it's not being accessed.

My more pressing issue right now is speeding up fontconfig.
My OS is Perception.
Post Reply