One file - many users (program instances) should write to it

[Wanderer82]

Hello, it's me again. After a while of not coding anything I wanted to continue my "little" app which works like a shop where different users can choose products they need (want to order), put them into a shopping cart and then send the order to the administrator.

For the products I use text files, one file for each product, in which, among other things, the quantity of products in stock and the quantity of products that are scheduled (until the order is quit by the administrator) are stored. So whenever a customer sends an order, the quantity of "scheduled products" needs to be changed in the product's text file. As this software can be used by more than one user at the same time, it is possible (although very very unlikely) that two or more users send an order at the exact same time which would lead to a conflict (race) when writing to the product's file(s).

I have already gone through the web but didn't find an easy solution to this problem so as I have already gotten many helpful advices in this forum, I'll try my luck

What I was first thinking about was a solution using a "lockfile" which is written before accessing the file and after having checked if another lockfile is already in the directory. After creating the lockfile, another test for other lockfiles would be performed. There are two problems: a) If the lockfile isn't deleted because of any issues during the execution of the programme, I have a problem. b) If both users send the request (write the lockfile) at the exact same time (and I don't even know what small time spans we're talking about here), it might also lead to the problem that one "process" might think that there is no other lockfile in the directory. I'm not that good at logical thinking, but I wrecked my head about this and I'm still not sure whether this problem is actually solvable with a kind of lockfile or not.

Any easy solution is very much appreciated.

Thanks a lot,
Thomas

[PB]

My 2 cents, this is probably not a realistically possible, you will end re-inventing a sharable database. Use a database instead as is done in such cases.

[Nunki]

Best beware for easy solutions, they tend to bite you back in the long run.
Go for a real database like Postgresql or MySql, or even MS sqlserver. SQlite is not an option for it too uses 1 datafile and has issues with simultaneous writes as has MS access.
It may seem to you as overkill but the design problems you're trying to overcome have been solved in a database system. Thousands of updates/inserts are managed by the database engine for you. In that way you have an elegant solution and less worries.

happy coding,
Nunki

[Wanderer82]

I have heard about a database solution but have put much effort in my code already. I'm a complete beginner and actually my expectations aren't very high for this program. Also this is written for - if it ever will be more than just a personal play-around - a school. So for teachers who can order material for their classes like ring binders, pencils and so on. So the probability that there are two users at exactly the same time trying to send an order is almost zero, but still this problem should be taken care of somehow. So, are you guys saying, that there is no other solution (without tons of additional and difficult code) to this problem using C++ only without any additional database like MySql?

Maybe @doublemax You have helped me a lot already in the past...

[PB]

I'm a complete beginner

This actually may be one more reason to not try to tackle this non-trivial problem. ;)

[Wanderer82]

Hm, probably you're right. But I just had a great idea: I'll just have the scheduled products written to extra files, so each order writes an individual file. If somone wants to check the scheduled products there will only be read accesses to all the orders, after which the scheduled products are calculated. That's actually quite a smart solution, or did I miss something?

[doublemax]

I'll just have the scheduled products written to extra files, so each order writes an individual file. If somone wants to check the scheduled products there will only be read accesses to all the orders, after which the scheduled products are calculated. That's actually quite a smart solution, or did I miss something?

It's easy to say now, but that would have been my suggestion

However: You say you're a complete beginner, but you're probably less of a beginner now, compared to when you started. Using an embedded database, e.g. sqlite with wxSQLite3 is not *that* hard. You should play around with that a bit in a stand-alone sample, and once you got a grip on it, you can decide how hard it would be to use it in your main application.

[Wanderer82]

Hi Doublemax

Thank you for chiming in.

Well, I haven't been coding for the last two years so a lot of what I learned isn't really right on my mind anymore.

But anyways, today I tried a simple piece of code using pipes to communicate between different processes. What I've learned so far is, that if I create a pipe, another process cannot create the same pipe anymore until it has been closed again. So I could use this for writing to the same file from two users. But my question is: What about if both processes try to create the same pipe at exactly the same time? Or is the creation of pipes atomic, so that the creation can never be at exactly the same time in another process? This solution seems somehow too easy to be true

Thanks,
Thomas

Well, in the meantime I realized that there is another problem. As the programm will be on a server drive and the users are logged in using different user names or they can even use the VPN connection to start a program, there can be more than one pipe for each user. But I guess there should be a solution to that?

[doublemax]

You want to (ab-)use the pipe as a file locking tool? Sorry, that's just horrible.

If you want to pursue the method of multiple processes writing to the same file, experiment with the "write_excl" file open mode of wxFile:
https://docs.wxwidgets.org/trunk/classw ... 879475fe3f

I've never used it, but it sounds like it could do what you need.

[Wanderer82]

I've shocked you So, why is using a pipe to make sure that only one instance of my program can change a file, horrible?

Okay, your suggestion sounds interesting. Although, the "write_ecl" open mode only works when creating a new file not for already existing ones. But maybe it might work anyway, as I wrote a temporary file anyway and then replaced the original one with that one, renaming it... I'll have to look into this possibility But I have a question: What exactly does it mean that the file is opened atomically? Does that mean that, if two processes try to create the same temporary file at "the same time", thre is no risk of a race condition - so I can be sure that only one process will be able to create the file while the other one will get an error?

But actually I don't understand. In the programming world, this problem shouldn't be so seldomly seen, am I wrong? I mean, you won't always use or need a database to get into a situation of having to write into one file by more than one process. So there should be a "standard" solution to this question... but then again, I'm just a beginner

EDIT:

Sorry, I have another question. I'm fiddling around with your suggestion. The problem is, if I try to access a file which is already opened by wxFile, I get an error message during runtime which I don't want to see. I only want it to return to the program that it is not allowed to access the file at the moment and sleep for some milliseconds. I looked inside the "file.cpp" and found the error message - well, I thought it was this one. I deleted it but it still keeps coming. Where can I change the behavior of wxWidgets classes?

EDIT 2:

I might have found another solution which I have already stumbled upon once. Actually it's also maybe a bit of an abuse but it comes very handy. So what I'm doing is, I use the "CreateFile"-Function from the "windows.h" library. This function opens a file atomically while restricting any other access before its handle is closed again. Now I wouldn't use the function to write to the file but as a guard. So there is kind of a "lockfile" which is always in the directory (doesn't have to be created or deleted, that's the good thing). If someone wants to write to a product, the program opens this file using CreateFile and while this handle / this file is open, it can rewrite the actual product file while the other program has to wait until the handle to the "lockfile" is closed. Like this I can go on using my code using ofstream which is easier than using CreateFile to write a file and I don't have to rewrite the code.

[doublemax]

What exactly does it mean that the file is opened atomically? Does that mean that, if two processes try to create the same temporary file at "the same time", thre is no risk of a race condition - so I can be sure that only one process will be able to create the file while the other one will get an error?

Yes.

So there should be a "standard" solution to this question..

Not exactlly my area of expertise, but i would guess that at the lower level, they all depend on the file locking mechanism of the OS. AFAIK with the "write_excl" open mode, this is one of the very few places where this is exposed to wxWidgets.

But in general, this is not a trivial task, especially if you don't want to lose performance. E.g. here is described how SQLite does it: https://www.sqlite.org/lockingv3.html

The problem is, if I try to access a file which is already opened by wxFile, I get an error message during runtime which I don't want to see.

Temporarily create an instance of wxLogNull to suppress the error message.
https://docs.wxwidgets.org/trunk/classwx_log_null.html

I use the "CreateFile"-Function from the "windows.h" library. This function opens a file atomically while restricting any other access before its handle is closed again.

Using lock files is not uncommon. E.g. the wxSingleInstanceChecker implementation under Linux uses one.

[Wanderer82]

Thanks a lot for your answer... so my way of using CreateFile seems to be fine as long as I need the program to work in Windows only, which in my case is the case

Thanks for the hint to suppress log messages. That might be very helpful to know.

I think the explanation of how SQLite does it, is a bit too difficult for me, but as I understand from just a glance it basically also uses a locking mechanism which is atomic.

Another question: Is it possible to intentionally simulate a case where two programs / instances try to write to the same file at the same time (the closest possible) to simulate what might happen if a lock(file) wasn't atomic. Whenever in a program I write a command twice, the program will process the commands sequentially, never simultaneously really, right?

[Nunki]

Hi Wanderer82,

I'm pretty sure the first instance who opens the file for writing (CreateFile) will succeed, the second will get a 'locked' error while opening and an invalid file handle.
However, reading this post I get the funny feeling you have spent a lot of time searching for a bypass of a commonly known problem with files, while you could have solved it by now with the database solution. It may seem complicated but it isn't. Installing PostgreSQL on a windows server is quite simple. Gaining access by means of the API (C++) is not that difficult. You can easily find lots of examples haw to launch an SQL query and fetching the results. There are even libraries that does the hard work for you.
And in return you get an application that is more stable and has options for future features.

Named pipes - and sockets - are a conduit, a means of processes or even machines to communicate with each other (sending messages, predefined and well known to both parties). They're not used for locking mechanisms. For that you have lockfiles and mutexes and semaphores.

Once again, you are building an application. you should focus on the functionality your user needs. Everything below that level you can solve with the right library or software. Like a database and a database layer.

happy coding,
Nunki

[Wanderer82]

Hello Nunki

You know, when I first started this project / programme, I was really just playing around. I don't write programs for a living, just for fun, so I was really a beginner, which I actually still am a bit, I would say. I have a lot of half-knowledge, I guess, and some of my code was more like "trial and error" sometimes. So I was quite happy and proud of being able to generate a "database" for these products with even different options and suboptions available. The code is quite long and I've put a lot of time and effort in it. So when I realized that there's another "easier" option, I just wanted to stick with my solution and try to solve every problem that would come up in a way which seemed possible to me. Now I've written my more or less complicated nested functions to read and write the product infos and it's just no fun for me to change all of them. That's the main reason, I'd like to stick with what I have. I know, I'm a bit of a strange guy When the program is finished, I might have the motivation to rewrite it using a database like SQLite to enlarge my knowledge, who knows.

[doublemax]

Another question: Is it possible to intentionally simulate a case where two programs / instances try to write to the same file at the same time (the closest possible) to simulate what might happen if a lock(file) wasn't atomic. Whenever in a program I write a command twice, the program will process the commands sequentially, never simultaneously really, right?

You can either use multiple threads, or - the most realistic test - start multiple instances of your (test-) application.