Cross platform embedding of resources

If you have a cool piece of software to share, but you are not hosting it officially yet, please dump it in here. If you have code snippets that are useful, please donate!
Post Reply
ItFinallyWorks
In need of some credit
In need of some credit
Posts: 4
Joined: Fri Nov 09, 2007 2:39 am

Cross platform embedding of resources

Post by ItFinallyWorks » Thu Jan 17, 2008 9:52 pm

So you have written your program and it has lots of features. It has help, it has lots of icons, it has po catalogs for all those translations. But all these features add potentially lots of files to your program and you would really like to go back to the simple time when there was just one file - the executable.

On windows, this can be solved since you can embed resources in the executable. On linux though, you still have lots of files. Here's how to embed arbitrary resources in an executable (cross platform compatible too).

Starting from http://www.wxwidgets.org/wiki/index.php ... PNG_Images, I had the idea that something like this is possible, but this still doesn't allow just anything to be embedded.

So, here's the process:
1. Take all your resources and make a zip file out of them (it can be more than one, but one is more convenient). For example, I'll call my zip file Example.zip
2. Using the c version of the program listed on that page, (http://www.wxwidgets.org/wiki/index.php ... der_guards) convert your zip file into c code. For example mine might be called Example.zip.h
3. #include the zip file in one of the first classes that executes. For example, as part of the class that inherits from wxApp, I use the code

Code: Select all

#include "Example.zip.h"
4. In the class where you #included the zip, somewhere early in the startup sequence of your program, add the following lines

Code: Select all

    wxFileSystem::AddHandler( new wxArchiveFSHandler );
    wxFileSystem::AddHandler( new wxMemoryFSHandler );
These lines enable your program to open zips and use a virtual memory file system
5. After those lines of code add something like the following

Code: Select all

    wxMemoryFSHandler::AddFileWithMimeType( _T("Example.zip"), Example_zip, sizeof(Example_zip), _T("application/zip") );
Of course, substitute your file name where appropriate. What this does is add that zip file (coded in such a way that c can understand it) into a virtual memory file system as the file Example.zip (it doesn't have to have the same name as the original zip file).
Note that according to the documentation, this call only works in wx 2.8
6. Now you can use wxFileSystem to access the contents of the zip. The path is memory:Example.zip#zip:someDirectoryInZip/someFile

A few examples:
Using the wx help system, to add a help file named HelpFile.hhp located in the Documentation directory of the zip file, do something like

Code: Select all

wxHtmlHelpController*  helpSystem = new wxHtmlHelpController( wxHF_DEFAULT_STYLE, this );
 helpSystem->AddBook( _T("memory:Example.zip#zip:Documentation/HelpFile.hhp") );
Then just use the help system like you normally would.


Getting an image out is a little trickier but here's how
Assuming the image is called image.png and is located in the icons directory,

Code: Select all

wxImage theImage;
wxFileSystem fs;
wxFsFile* imageFile;

imageFile = fs.OpenFile( _T("memory:Example.zip#zip:icons/image.png") );

if( imageFile != NULL )
{
    theImage.LoadFile( *imageFile->GetStream(), imageFile->GetMimeType() );

    //do something with the image
}
else
    //the image does not exist, error handling

Basically, if the wx code uses wxFileSystem, you can just pass the path. If not, you might have to resort to some trickery like using an input stream (like the image example). In the worst case, you could just unpack a file out of the embedded zip into a temp folder or somewhere else by 1) getting a stream, as in the image example, then 2) using the function wxTransferStreamToFile to write the stream out to a file.

One limitation of this method is that you cannot change any of these files - they are compiled into the executable, so changing them in the executable is a somewhat difficult task.

Jorg
Moderator
Moderator
Posts: 3971
Joined: Fri Aug 27, 2004 9:38 pm
Location: Delft, Netherlands
Contact:

Post by Jorg » Fri Jan 18, 2008 8:49 am

Looks great .. Thanks for your contribution!
Regards,
- Jorgen
Forensic Software Engineer
Netherlands Forensic Insitute
http://english.forensischinstituut.nl/
-------------------------------------
Jorg's WasteBucket
http://www.xs4all.nl/~jorgb/wb

mispunt
Experienced Solver
Experienced Solver
Posts: 59
Joined: Tue Oct 19, 2004 3:23 pm
Location: Ede, Holland

Post by mispunt » Thu Jan 24, 2008 3:13 pm

I never did it this way... I always used the "cat" style.

Code: Select all

cat example.exe.tmp example.zip > example.exe
The cat program is originally a unix program, but there is a windows version for it.

btw. this only means you don't have to compile it within you're program, and don't have to recompile the whole program when you change something in the zipfile only...
OS: win XP pro
Compiler: MingW
wxWidgets version: 2.6.2

ItFinallyWorks
In need of some credit
In need of some credit
Posts: 4
Joined: Fri Nov 09, 2007 2:39 am

Post by ItFinallyWorks » Thu Feb 21, 2008 1:49 am

I never did it this way... I always used the "cat" style.
wxwidgets:
cat example.exe.tmp example.zip > example.exe
The cat program is originally a unix program, but there is a windows version for it.
I see how that works, but how do you get the data out from within your program? Do you open your exe with wxFS?

Thanks

mispunt
Experienced Solver
Experienced Solver
Posts: 59
Joined: Tue Oct 19, 2004 3:23 pm
Location: Ede, Holland

Post by mispunt » Wed Mar 19, 2008 11:51 am

ItFinallyWorks wrote:Do you open your exe with wxFS?
Yes you do, simple as that :)
OS: win XP pro
Compiler: MingW
wxWidgets version: 2.6.2

Utensil
Moderator
Moderator
Posts: 423
Joined: Sun Feb 03, 2008 11:38 am
Location: China

Post by Utensil » Wed Mar 19, 2008 3:43 pm

mispunt wrote:
ItFinallyWorks wrote:Do you open your exe with wxFS?
Yes you do, simple as that :)
Very smart! But how do you locate the beginning of the zip?

-Utensil
In fascination of creating worlds by words, and in pursuit of words behind the world.

On Github: http://utensil.github.com
Technical Blog in Chinese: http://utensil.iteye.com/

upCASE
Site Admin
Site Admin
Posts: 3176
Joined: Mon Aug 30, 2004 6:55 am
Location: Germany, Cologne

Post by upCASE » Wed Mar 19, 2008 4:25 pm

Hi!
Nicely done!

It's a nice way to handle stuff like on the mac where a bundle is nothing more than a directory where you can place your stuff in.
Utensil wrote:Very smart! But how do you locate the beginning of the zip?
Not confirmed as I didn't try, but wouldn't wxFileSystem::FindFirst() help?
OS: OpenSuSE, Ubuntu, Win XP Pro
wx: svn
Compiler: gcc 4.5.1, VC 2008, eVC 4

"If it was hard to write it should be hard to read..." - the unknown coder
"Try not! Do. Or do not. There is no try." - Yoda

mc2r
wxWorld Domination!
wxWorld Domination!
Posts: 1195
Joined: Thu Feb 22, 2007 4:47 pm
Location: Denver, Co
Contact:

Re: Cross platform embedding of resources

Post by mc2r » Wed Mar 19, 2008 6:14 pm

Nice work, not trying to detract from it at all I just wanted to clarify one point.
ItFinallyWorks wrote:On windows, this can be solved since you can embed resources in the executable. On linux though, you still have lots of files.
it is possible to embed resources in an elf binary(linux executable). objcopy for instance will create a .o which can be linked to your elf executable like any other.

EDIT:
Also, I believe binutils and objcopy are available for OS X and Mach-O but have no way to verify if this works.

-Max

ItFinallyWorks
In need of some credit
In need of some credit
Posts: 4
Joined: Fri Nov 09, 2007 2:39 am

Post by ItFinallyWorks » Thu Mar 20, 2008 6:43 pm

it is possible to embed resources in an elf binary(linux executable). objcopy for instance will create a .o which can be linked to your elf executable like any other.
That's interesting. I didn't know you could do that. I know its fairly common on windows (that's how you get those nice icons on the exe), but I hadn't seen it done on Linux. It seems like the plain exe as is common on Windows doesn't really exist on Linux - everything has to be installed.

ItFinallyWorks wrote:
Do you open your exe with wxFS?
Yes you do, simple as that
It's a nice way to handle stuff like on the mac where a bundle is nothing more than a directory where you can place your stuff in.
Utensil wrote:
Very smart! But how do you locate the beginning of the zip?
Not confirmed as I didn't try, but wouldn't wxFileSystem::FindFirst() help?
Ahh, I see how this might work now. Interesting. I wonder if there is an advantage to doing one over the other. Memory usage maybe? Or just ease of use. One thing I can see about the cat the zip file is that it requires a modification to the build process, wheras mine works with the standard build process. On the other hand the cat the zip file allows quick changes to the zip file and mine requires a rebuild.

Interesting discussion - thanks for the information.

mc2r
wxWorld Domination!
wxWorld Domination!
Posts: 1195
Joined: Thu Feb 22, 2007 4:47 pm
Location: Denver, Co
Contact:

Post by mc2r » Thu Mar 20, 2008 8:01 pm

ItFinallyWorks wrote:
it is possible to embed resources in an elf binary(linux executable). objcopy for instance will create a .o which can be linked to your elf executable like any other.
That's interesting. I didn't know you could do that. I know its fairly common on windows (that's how you get those nice icons on the exe), but I hadn't seen it done on Linux.
I didn't know about this until recently either. I stumbled across it while looking up some stuff for a windows app that embeds resources this way in its exe. And then this post showed up with perfect timing to share ;)
ItFinallyWorks wrote:It seems like the plain exe as is common on Windows doesn't really exist on Linux - everything has to be installed.
I'm not sure what you mean by this? To me linux binaries seem much more flexible and don't need to be installed as you can usually run them from any directory. No registery or anything like that.

-Max

ItFinallyWorks
In need of some credit
In need of some credit
Posts: 4
Joined: Fri Nov 09, 2007 2:39 am

Post by ItFinallyWorks » Wed Mar 26, 2008 1:30 am

mc2r wrote:
ItFinallyWorks wrote:
it is possible to embed resources in an elf binary(linux executable). objcopy for instance will create a .o which can be linked to your elf executable like any other.
That's interesting. I didn't know you could do that. I know its fairly common on windows (that's how you get those nice icons on the exe), but I hadn't seen it done on Linux.
I didn't know about this until recently either. I stumbled across it while looking up some stuff for a windows app that embeds resources this way in its exe. And then this post showed up with perfect timing to share ;)
ItFinallyWorks wrote:It seems like the plain exe as is common on Windows doesn't really exist on Linux - everything has to be installed.
I'm not sure what you mean by this? To me linux binaries seem much more flexible and don't need to be installed as you can usually run them from any directory. No registery or anything like that.

-Max
What I mean by this is that it is fairly common to see standalone windows executables - many come that way. Most Linux executables come as a .deb or a rpm or source. You might be able to run them standalone, but they, at the least, need to be extracted. At worst, they need to be recompiled/edited to fix absolute directories. Having a package managed OS does have its downsides since authors can expect their data and libraries to be in fixed locations.


I'm not criticizing the binary format by any means. Linux is by far the most flexible OS I have encountered. Its more of the establishment I criticize (like why does my application have to be mixed with every other program on my hard drive and spread across multiple distant folders, with multiple possibilities for each (like /bin, /usr/bin, /usr/local/bin)). Too bad stuff like http://www.gobolinux.org/ isn't more common.

Belgabor
I live to help wx-kind
I live to help wx-kind
Posts: 173
Joined: Mon Sep 25, 2006 1:12 pm

Post by Belgabor » Fri Mar 28, 2008 1:59 am

It is as possible on linux as on windows to create a self-sufficient "exe". It's just usually not done. The reason is (probably) that linux traditionally runs on a lot of differet platforms, while Windows is pretty much x86 only.

Post Reply