OSDev.org

Someone who is working on my project is trying to port FreeType to it. He succeeded (within his local git repo), but the issue is that FreeType is an order of magnitude larger than the rest of the project (250K lines vs. 15K lines), and it seems like a bad idea (partially for space/bandwidth reasons) to effectively copy the whole FreeType source tree into the project, even into a separate directory. In addition, we're not sure whether the main LICENSE file needs to be be amended with something to say that parts of the project are not of the same license.

These are the two possible alternatives I've come up with: 1. make a patch for the current FreeType source tarball and only include that (the changes are minor and mostly to the build system), and make developers manually download, unpack, and patch it before they can test the system; and 2. include only the compiled static library of FreeType (compiled against the C library, which is relatively stable.) Both of these seem sub-optimal, however.

What do people usually do in this situation?

It depends a little on the patches. The ideal outcome is that the patches are fed back to the freetype project so that freetype can include Rhombus in its list of supported operating systems.

That's probably a little difficult until everybody is using Rhombus, so in the meantime you need to maintain the patches outside freetype. They need not (perhaps should not) however be part of the Rhombus project. Put the patches in a separate project. Maybe you need to create the rhombus-ports project.

gerryg400 wrote:It depends a little on the patches. The ideal outcome is that the patches are fed back to the freetype project so that freetype can include Rhombus in its list of supported operating systems.

In this particular case, the patches don't really change the internals of FreeType, just some environment variables in the Makefile to point it to the cross-compiler, so I don't think it's worth giving back to FreeType. These changes will probably be reversed when I get the system self-hosting.

gerryg400 wrote:That's probably a little difficult until everybody is using Rhombus, so in the meantime you need to maintain the patches outside freetype. They need not (perhaps should not) however be part of the Rhombus project. Put the patches in a separate project. Maybe you need to create the rhombus-ports project.

The issue is that FreeType is likely to become an important part of the base system (I'm going to switch the terminal driver to using it pretty soon, for example,) so it will be needed for testing. I'd prefer not to have potential developers have to download two separate repositories just to boot the system. For less-essential ports, that is definitely a good idea though.

For porting 3rd party software, including patches (..that can't be sent upstream) a some build glue seems to be cleaner in the long run.

What we're doing in tyndur is basically what has been suggested here: We have a separate repository which contains a directory for each port. In this directory there are patches and a small shell script that configures and compiles the whole thing (actually, just the implementation of some shell functions, it's included from a wrapper script that contains the common things).

As an example, this is what the freetype directory looks like: http://git.tyndur.org/?p=lbuilds.git;a= ... type/2.4.1

LOL... Gentoo ebuilds for the win.

(No, really. I've used that approach myself on more than one occassion. Nice to see I'm not alone. )

There seems to be a reason why it's used.

The only thing that I still find a bit nasty is creating the patches. When unpacking the thing, we do set up everything with quilt so that you can just fix things and update the patches while trying to get it built, but I always forget to add the files to the patch first. Need to get something git based some time...

Not quite sure I understood what it is that troubles you.

I've got this function in the wrapper for applying the patches:
I.e., I've got icu/files/3.6/allinone__allinone.sln.patch to patch ./allinone/allinone.sln in the ICU source tree. Keeps all patches for a given upstream source tarball in one directory, while at the same time showing which file it's meant for in the filename. Worked for me so far.

When a new version is released by upstream, I run the old patches against the new sources and fix what's broken. Worked so far for me, but then again it has always been only a handful of patches against two or three upstream projects only, so nothing major.

Kevin wrote:What we're doing in tyndur is basically what has been suggested here: We have a separate repository which contains a directory for each port. In this directory there are patches and a small shell script that configures and compiles the whole thing (actually, just the implementation of some shell functions, it's included from a wrapper script that contains the common things).

As an example, this is what the freetype directory looks like: http://git.tyndur.org/?p=lbuilds.git;a= ... type/2.4.1

Since you're using Git, why not a repository for each port? Certainly, when I get around to it my plan is. This has the advantage, IMO at least, that you can just pull upstream changes into the repository and merge them. It seems to be the strategy that a number of Linux distributions (such as Debian, Gentoo and Android) often take

Owen wrote:This has the advantage, IMO at least, that you can just pull upstream changes into the repository and merge them.

Isn't this assuming that upstream is using git, too?

Solar wrote:

Owen wrote:This has the advantage, IMO at least, that you can just pull upstream changes into the repository and merge them.

Isn't this assuming that upstream is using git, too?

No. Git has git-svn, for working with upstream SVN repositories, plus externals support for working with CVS as well (IIRC). Mercurial has support for all of the above plus working with Git; I know Bazaar has plugins for interworking with Git and Hg.

Plus, quite a lot of SVN-based projects are setting up Git and/or Hg mirrors of their code these days; LLVM (Git) and OpenTTD (Git and Hg) come to mind.

Working with a non-native upstream is often slightly different from working with a native one, but in general you can still work in the same manner, and most of the issues occur when you try to push changes upstream (e.g. doing a git-svn dcommit to an SVN repository requires rewriting the history of the Git repository, because the SVN metadata format isn't expressive enough) . This is one of the killer features, IMO, of a DVCS; branching and merging are effortless. Some centralised systems (e.g. per my understanding, Perforce) get close, but none of then manage to do it as effectively or efficiently as a DVCS.

At the end of the day, use whatever system you want; but for this kind of situation, a good DVCS makes things much easier

Two reasons come to mind why using a git repository for every port might not be what I want: The first has already been mentioned by Solar. It restricts you to projects using git and/or SVN. CVS didn't work well when I tried it, hg can export to git, but git can't natively clone hg repositories, and then you still have projects that only come as a tarball because their author has never heard of or is scared by version control.

The second reason is that I'm really not interested in the whole project history. I want to port a specific version of the application to my OS, and I don't want to waste my disk space with umpteen ancient versions of the same project. It's not like I'm going to do any serious development on those projects so that the history would help me. The patches are usually very limited, like commenting out stuff that isn't supported by our libc yet, and as such are very easy to port forward or even rewrite.

Not quite sure I understood what it is that troubles you.

Applying patches isn't a problem. Even forward porting them is doable manually. What I don't like is the way to create them in the first place. The usual procedure for getting some software ported is that you create the lbuild file and just start building. At some point, the build will break because the libc is lacking something or just because it isn't quite Linux. So you try and fix that and want to make a patch out of your change.

The traditional way would be to have create a clean copy of the tree, modify one, diff -urN. Hope that you don't have changes in generates files that you would have to filter out manually. After the next build failure you put in another hack, diff -urN again to update the patch. Kind of works for a big monolithic patch, but it isn't very convenient. It starts to become completely unmanageable by hand if you want cleanly separated patches for different changes.

Basically, what you really want is some VCS kind of thing. You hack up the code until it works, then you commit and get the patch. quilt comes very close to this, except that it's a bit backwards: First you need to tell it that you want a new patch and which files you're going to modify, and then you modify the files. If you don't tell it which file you'll edit it doesn't generate the patch (because obviously it's still doing the copy of the clean version internally). However, if you manage to do that properly, quilt works really well, and it's super easy to update patches etc. It's just that I always forget to do it in the right order.

Kevin wrote:Two reasons come to mind why using a git repository for every port might not be what I want: The first has already been mentioned by Solar. It restricts you to projects using git and/or SVN. CVS didn't work well when I tried it, hg can export to git, but git can't natively clone hg repositories, and then you still have projects that only come as a tarball because their author has never heard of or is scared by version control.

The second reason is that I'm really not interested in the whole project history. I want to port a specific version of the application to my OS, and I don't want to waste my disk space with umpteen ancient versions of the same project. It's not like I'm going to do any serious development on those projects so that the history would help me. The patches are usually very limited, like commenting out stuff that isn't supported by our libc yet, and as such are very easy to port forward or even rewrite.

<--- snip --->

Basically, what you really want is some VCS kind of thing. You hack up the code until it works, then you commit and get the patch. quilt comes very close to this, except that it's a bit backwards: First you need to tell it that you want a new patch and which files you're going to modify, and then you modify the files. If you don't tell it which file you'll edit it doesn't generate the patch (because obviously it's still doing the copy of the clean version internally). However, if you manage to do that properly, quilt works really well, and it's super easy to update patches etc. It's just that I always forget to do it in the right order.

If you can't inter-operate with the upstream repository, then you're in no worse situation by maintaining one of your own for generating packages from than you were before.

Kevin wrote:The traditional way would be to have create a clean copy of the tree, modify one, diff -urN. Hope that you don't have changes in generates files that you would have to filter out manually.

Don't diff against the distribution tarball, diff against the VCS checkout...

NickJohnson

NickJohnson wrote:Someone who is working on my project is trying to port FreeType to it. He succeeded (within his local git repo), but the issue is that FreeType is an order of magnitude larger than the rest of the project (250K lines vs. 15K lines), and it seems like a bad idea (partially for space/bandwidth reasons) to effectively copy the whole FreeType source tree into the project, even into a separate directory. In addition, we're not sure whether the main LICENSE file needs to be be amended with something to say that parts of the project are not of the same license.

These are the two possible alternatives I've come up with: 1. make a patch for the current FreeType source tarball and only include that (the changes are minor and mostly to the build system), and make developers manually download, unpack, and patch it before they can test the system; and 2. include only the compiled static library of FreeType (compiled against the C library, which is relatively stable.) Both of these seem sub-optimal, however.

What do people usually do in this situation?

You have the 3rd alternative - not to use FreeType and use another renderer of fonts.
Few month ago FreeType was ported to KolibriOS by Sorcerer. But we come to agreement that FreeType is too big to be used in KolibriOS and Sorcerer used his experience to make his own renderer. It is not yet usable because we yet didn't decided what API should the library has, but you can participate in the discussion.
I don't know if you can understand it after translate of google.