我使用Websphere MQ，它默认带有“基于文件的队列”。 因此每个队列通常在磁盘上有一个物理文件。
因此，对于这个Websphere MQ队列，我们有多个读取器和写入器....我假设读取和写入之间会有一些“毫秒锁定”开销，但我保证它仍然比使用DB2等完整数据库更快。
现在我已经使用SQLite实现了一个队列，它可以在WAL模式下愉快地工作。
我的问题是你真的认为在WebSphere MQ等产品或基于SQLite的队列使用的基于文件的队列之间的SPEED存在巨大差异吗？ 毕竟两者都是基于文件的，并且SQLIte是纯ANSI C这一事实对速度有利。
SQLIte确实提供了一些额外的功能，如“更新钩子”等......以及其他许多功能。
我想我想知道的是，如果您要在C中实现高速持久队列，您是否会以类似于WebSphere MQ的方式执行此操作，并在单个文件中将消息放入不同的偏移量等，或者您是否会使用SQLIte在WAL模式？
谢谢您的帮助 ;-）
I use Websphere MQ and it comes by default with a "file based queue". So each queue typically has ONE physical file on disk.
So lets say for this Websphere MQ queue we have multiple readers and writers....I assume there will be some "millisecond locking" overhead between reads and writes, yet I guarantee it will still be quicker than using a full database like DB2 etc.
Now I have implemented a queue using SQLite and it works in Production happily in WAL mode.
My question is do you really think there is a huge difference in SPEED between the file based queue used by products like WebSphere MQ or the SQLite based queue? Both are file based after all and the fact that SQLIte is pure ANSI C does it well for speed.
SQLIte does offer some extra functionality like "update hooks" etc....and numerous others.
I guess what I would like to know is if you were going to implement a high speed persistent queue in C would you do it in a similar way to WebSphere MQ and have messages in different offsets etc in a single file, or would you use SQLIte in WAL mode?
Thanks for the help ;-)
最满意答案
您对示例的约束越多，您可能就越能够使数字看起来像您想要的任何内容。 一个应用程序的单个队列？ 如果这是您唯一的要求，那么您有很多选择。
但是看一看WAL模式的一些限制。 检查点等待读者完成。 因此，对于更多的读者来说，检查点越难，检查点就越具有破坏性。 如果WAL文件变大，那么读取性能会降低，因此您无法在繁忙的系统上进行延迟检查点并保持高性能。
因此，问题“ 如果你要在C中实现高速持久化队列，你会以类似于Websphere MQ的方式实现它，并在单个文件中使用不同的偏移量等消息，还是在WAL模式下使用SQLIte？ ”没有捕获所有要求或考虑因素。 随着并发用户数量的增加，流程架构开始掩盖了您所询问的存储方法的差异。 WMQ可以处理数千个并发读取器和写入器，其中非常大的日志文件具有相当平坦的性能曲线，而WAL文档表明性能与WAL文件的大小成比例，即性能曲线随着WAL变大而趋于下降。
如果我要在C中实现高速队列？ 如果您的意思是我选择哪种现有产品，我会选择经过二十年调整和优化的产品，每年投入数百万美元进行研发，并专注于排队。 如果通过“实施”你的意思是我如何从头开始编写一个新的排队引擎，那么我会受到那些专注于排队很长时间的产品的影响，并试图重用他们的解决方案对于非平凡的问题。 数据库和排队引擎不会偶然到达各自的存储架构。 一个针对队列进行了优化，一个针对数据库语义进行了优化，如果扩展范围以包括除WMQ和SQLite之外的数据库和排队引擎，则这些类别通常都是正确的。
完全披露：我在WMQ工作了20年的大部分时间，最近加入了IBM的产品管理团队。 我可能有点偏颇，但我试图专注于这里的技术，而不是一个下意识的“我的产品比你的产品更好”的反应。 随意表示同意投票，不同意投票。 如果得票太多，我会撤回答案。
The more constraints you place on the example, the more you would probably be able to make the numbers look like whatever you wanted. A single queue with one app? If that's your only requirement then you have lots of choices.
But take a look at some of the restrictions on WAL mode. Checkpoints wait for readers to complete. Therefore more readers you have the harder it is to checkpoint and the more disruptive checkpointing becomes. If the WAL file gets large then read performance degrades so you cannot do lazy checkpointing on a busy system and remain performant.
So the question "if you were going to implement a high speed perstent queue in C would you do it in a similar way to Websphere MQ and have messages in different offsets etc in a single file, or would you use SQLIte in WAL mode?" doesn't capture all the requirements or considerations. As the number of concurrent users scales, the process architecture begins to overshadow the differences in storage approaches you are asking about. WMQ can handle thousands of concurrent readers and writers, with very large log files with fairly flat performance curves whereas the WAL docs state that performance is proportional to the size of the WAL file, i.e. performance curves trend down as the WAL gets large.
If I were going to implement a high speed queue in C? If you mean which existing product do I pick, I'd go for the one that has had twenty years of tuning and optimization with millions of dollars in R&D invested each of those years and which focuses exclusively on queuing. If by "implement" you mean how would I do it if I were to write a new queuing engine from scratch, then I'd be heavily influenced by the products that have focused on queuing for a really long time and try to reuse their solutions to the non-trivial problems. Databases and queuing engines didn't arrive at their respective storage architectures by accident. One is optimized for queues and one is optimized for database semantics and if you expand your scope to include DBs and queuing engines other than WMQ and SQLite this holds generally true across the categories.
Full disclosure: I've been working with WMQ for most of the twenty years it's been around and recently joined its product management team at IBM. I might be a bit biased but I've tried to focus on the technology here and not a knee-jerk "my product is better than your product" reaction. Feel free to signify agreement with up-votes, disagreement with down-votes. I'll withdraw the answer if it gets too many down-votes.
更多推荐
发布评论