You are not logged in.

#1 2013-03-15 10:11:16

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Software catalog

Hi guys,

I'm working on OSM Software Catalog. It is improved version of the Software Tables.

Wiki.openstreetmap.org will be used as data source.

My question is how can I integrate it to OSM Wiki?

Catalog will consist of:

  • One HTML file

  • Few Javascript files

  • One JSON file. It will be automatically generated.

I'm just not sure if it can be easily converted to Wiki-doc format.

POC: http://dl.dropbox.com/u/62996911/OpenStreetMap.7z

UPD: Desktop application for Windows https://sourceforge.net/projects/osm-software-catalog/

Last edited by Dmytro Ovdiienko (2016-01-23 22:53:35)

Offline

#2 2013-03-17 15:34:30

saerdnaer
Member
Registered: 2011-12-04
Posts: 38

Re: Software catalog

Before starting coding, I would recommend to test, if existing mediawiki extension already provide the features you would like to have.

For example: On https://www.mediawiki.org/wiki/Extension:TableEdit it's mentioned that there is some support for http://datatables.net/ .

Last edited by saerdnaer (2013-03-17 15:34:54)

Offline

#3 2013-03-18 00:04:59

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

Thanks. I will take a look.
Looks like I have to find list of supported extensions. And way to contact Wiki administrator to install new extension.

Offline

#4 2013-03-18 12:02:20

saerdnaer
Member
Registered: 2011-12-04
Posts: 38

Re: Software catalog

I think you have to setup a test mediawiki and demonstrate that it works and there is an additional benefit. Maybe we should create a copy of the osm wiki where new extensions can be tested.

Regards,
Andreas

Last edited by saerdnaer (2013-03-18 12:38:50)

Offline

#5 2013-04-01 03:19:40

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

List of errors found during processing:
http://dl.dropbox.com/u/62996911/errors.7z

There are a lot of navigateRoute parameters. People do not follow specification.

Offline

#6 2013-04-01 09:48:40

saerdnaer
Member
Registered: 2011-12-04
Posts: 38

Re: Software catalog

Dmytro Ovdiienko wrote:

People do not follow specification.

It's a wiki. Just like the Openstreetmap Database itself. ;-)

And by the way: Which specification?

Offline

#7 2013-04-01 13:17:19

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

saerdnaer wrote:

It's a wiki. Just like the Openstreetmap Database itself. ;-)

Yeah.

saerdnaer wrote:

And by the way: Which specification?

A lot of applications. Each line of the "log" file contains name of application, error message related to the parameter, parameter itself and its value.

Offline

#8 2013-04-01 14:33:15

!i!
Member
Registered: 2009-11-28
Posts: 3,294
Website

Re: Software catalog

You might ask user:Tordanik before, he is working on improving his template bot as well. Please stay tuned, as he is currently very busy.


privater Account von KVLA-HRO-Mei

Offline

#9 2013-04-02 09:42:47

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

http://dl.dropbox.com/u/62996911/OpenStreetMap.7z
It is first proof-of-concept. Very limited functionality.

Offline

#10 2013-04-03 15:20:12

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

I've updated archive. Now I've achieved my goal to find Android application which supports offline routing.
1. Filter by Android
2. Sort by Offline Routing column

smile

Last edited by Dmytro Ovdiienko (2013-04-03 15:20:50)

Offline

#11 2013-04-05 09:34:22

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

saerdnaer wrote:

And by the way: Which specification?

Sorry, I'd read your question incorrectly. Specification is http://wiki.openstreetmap.org/wiki/Template:Software2

Offline

#12 2013-04-05 20:13:00

Tordanik
Moderator
From: Germany
Registered: 2008-06-17
Posts: 2,439
Website

Re: Software catalog

I've commented on the topic of unspecified template parameters via mail, but I'd like to describe another idea that may be relevant here.

There are several different projects parsing OSM wiki templates - TTTBot, UserGroupsBot, Taginfo, and an upcoming project by my regional OSM user group, to name only those I have had direct contact to. Therefore, I considered extracting the basic template parsing functionality into a common service serving a json representation of wiki templates.

This service is supposed to offer the following features:

  • make templates more machine-readable by hiding the details of the MediaWiki query api, and filtering out comments and other formatting details from the wiki pages themselves

  • provide a frequently updated view of the wiki by only parsing those pages that have changed since the last run

  • join the translations of a page with templates into a single json object containing all the localised strings

  • support both the case with just one template per page and multiple templates per page

  • provide common functionality such as constructing image thumbnail links, which is necessary for embedding wiki images outside the wiki

TTTBot is still lacking many of those features (and afaik none of the other examples I provided has all of them either), so the software catalogue would also profit from the implementation of this service. But the larger vision is of course to make it easier for all projects to access the OSM wiki, and to avoid reinventing the wheel each time.

To achieve this goal, the service is not supposed to contain any intelligence related to how the template content is being used, that would be the consumers' task. And it is supposed to require as little configuration as possible: It should figure out the type of the value (text, link, image, url, list, ...) from the value itself instead of requiring a full schema for each template.

I've even started writing it, but didn't get very far, and because I'm very busy this month I cannot work on wiki template parsing during that time period at all. I still wanted to to get your feedback on the idea and to ask you to maybe consider it for your future plans.

Offline

#13 2013-04-05 21:54:52

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

Tordanik wrote:

I still wanted to to get your feedback on the idea and to ask you to maybe consider it for your future plans.

I like your idea and I'm ready to use your JSON service instead of my JSON generator. We just have to discuss structure of the file.
On my future plans. I'm implementing Catalogue with my colleague. He is busy this week so it will be not so quick as I wish. My vision of this project I tried to explain in my first message of this thread. Other details we can discuss by mail or skype.

Please find my comments below.

Tordanik wrote:

serving a json representation of wiki templates

If json file would contain all the templates, it will be large. In this case we also have to implement filtering engine which will work on the server side.
Right now we have ~350 apps and size of Software2 template file is 220kb. Not a big but not a small. Gzipped file is 39kb.

Tordanik wrote:

provide a frequently updated view of the wiki by only parsing those pages that have changed since the last run

Agree.
What if provide link to JSON validator or generator so user will be able to validate his text in the same time and do not wait for bot?

Tordanik wrote:

provide a frequently updated view of the wiki by only parsing those pages that have changed since the last run
join the translations of a page with templates into a single json object containing all the localised strings
support both the case with just one template per page and multiple templates per page

Agree

Tordanik wrote:

provide common functionality such as constructing image thumbnail links, which is necessary for embedding wiki images outside the wiki

It will be great. I did not know how to get link to screenshot.

Tordanik wrote:

It should figure out the type of the value

I'm not sure if it is possible. I agree to replace links with "<a href='...'/>" but lists... Right now items of the list are delimited by semicolon and the Description parameter can have this character too.

Offline

#14 2013-04-09 23:25:20

Tordanik
Moderator
From: Germany
Registered: 2008-06-17
Posts: 2,439
Website

Re: Software catalog

Dmytro Ovdiienko wrote:
Tordanik wrote:

serving a json representation of wiki templates

If json file would contain all the templates, it will be large. In this case we also have to implement filtering engine which will work on the server side.
Right now we have ~350 apps and size of Software2 template file is 220kb. Not a big but not a small. Gzipped file is 39kb.

It is true, but originally that wasn't even my intent. What I want to do is making wiki parsing easier. So as a start, one would only replace any code accessing the MediaWiki API and searching for templates in the results with an access to my json representation of wiki templates.

Of course it would be nice if dynamic pages could be generated directly based that json, but as you say this might create additional requirements on the server - such as filtering. Do you think this is possible without semantic knowledge specific to each application?

What if provide link to JSON validator or generator so user will be able to validate his text in the same time and do not wait for bot?

I'm not sure if I fully understand this. Would you expect the user to paste the template code into that validator, or would that functionality become available somehow after saving the page?

Tordanik wrote:

It should figure out the type of the value

I'm not sure if it is possible. I agree to replace links with "<a href='...'/>" but lists... Right now items of the list are delimited by semicolon and the Description parameter can have this character too.

The idea here would be not to decide on one value type for each value, but to include all possible "parsings" for a given value. So in the case of a description with semicolon, you may end up with something like this:

"description" : {
"text": "offline router; good",
"list": ["offline router", "good"]
}

When using that json, the data consumer who thinks/knows that descriptions should be text will look for a "description" that has a "text", and simply ignores the rest. This moves the knowledge about what value types are expected to the data consumer, allowing the template->json translation to remain generic and easy to extend to more templates.

Last edited by Tordanik (2013-04-09 23:27:41)

Offline

#15 2013-04-10 00:13:34

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

Tordanik wrote:

Do you think this is possible without semantic knowledge specific to each application?

There are a lot of successful examples: Google Play, AppStore.
Anyway, it is just a database. We have to think how to design structure. To design structure we need list of all use cases and requirements. I will prepare requirements of Catalogue.

We can provide three options:

  • Option to parse Templates like how it works right now. We just have to add some samples and documentation. Maybe implement Template parser in Java?

  • Option to download complete new JSON file. The difference comparing to first option is this file is easy to parse.

  • Option to download part of the new JSON file using server side filtering.

All options will be implemented one by one in the order of the listing. User will choose suitable. It is like .NET/Java approach. You can start at any point of abstraction (File->Stream->Serializer).

I'm not sure if I fully understand this. Would you expect the user to paste the template code into that validator, or would that functionality become available somehow after saving the page?

I prefer "after saving page".
If wiki does not support triggers, we can add some button to regenerate JSON file.

When using that json, the data consumer who thinks/knows that descriptions should be text will look for a "description" that has a "text", and simply ignores the rest. This moves the knowledge about what value types are expected to the data consumer, allowing the template->json translation to remain generic and easy to extend to more templates.

I believe it is better to have strong typified fields. If field is of text type, consumer should use it as text. If field is of list type, consumer should treat field as list. Otherwise... it is hard to control software. What we need is template validator (to fix errors in the edit time) and JSON generator with option to generate JSON files of deprecated formats to support old consumers at least some time (in case of format changing). For example right now price field is list. OSM-JSON v1 will have price as list. However I'd like to have separated fields per currency per ApPStore. If we will change JSON structure we still will be able to generate OSM-JSON v1.

Last edited by Dmytro Ovdiienko (2013-04-12 13:54:13)

Offline

#16 2016-01-23 00:43:18

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

I did not find Web developer to implement my idea so I've implemented desktop application. Following is a link to download page:

http://sourceforge.net/projects/osm-software-catalog/

Application is still beta. It does not have all possible functionality. I will continue working on this app.

BTW, app can be used as validator for Tables::Software2 tags. It generates log file in the Data folder. There are a lot of warnings.

Last edited by Dmytro Ovdiienko (2016-01-23 08:50:16)

Offline

#17 2016-01-23 10:31:32

stephan75
Member
Registered: 2008-05-28
Posts: 2,756

Re: Software catalog

So are there plans to have OSM Software Catalog also as successor or replacement for the still-out-of-order TTTBot?

Offline

#18 2016-01-23 11:21:36

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

I would be happy to make OSM Software Catalog one of the official applications. But I think TTTBot should be fixed too. At least Linux users need it and those who do not want to run any software locally.

Offline

#19 2016-01-23 21:41:50

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

There is problem with price field.
0. Some apps are for free
1. Some applications are for free but some functionality is locked. However it is still possible to use them.
2. Other applications are for free but they have trial period.
3. Other applications are for free but maps are not free (or there is download limit)
4. Some apps are for free for Android but not for iPhone

I need some criterion to determine 0, 1 and 4  apps as freeware.... Right now "price" field is plain text field.

Last edited by Dmytro Ovdiienko (2016-01-23 22:55:03)

Offline

#20 2016-01-25 14:36:37

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

v0.0.5 is ready.
* Settings are persisted
* New fields are added to filter.
* Improve controls allocations to fit on small displays
* Minor changes in the interface
* Downgrade from .NET 4.5 to .NET 4.0 to be able to run on WinXP

Last edited by Dmytro Ovdiienko (2016-01-25 14:37:13)

Offline

#21 2016-01-27 00:00:00

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

v0.0.7
* Add accessibility support
* Add icon for Wiki button
* Set timestamps of *.wiki files based on page last edit time.
* Add 'Show status bar' command
* Sources are available on git server (Sourceforge)

Your comments and ideas are welcome.

Last edited by Dmytro Ovdiienko (2016-01-27 00:10:04)

Offline

#22 2016-02-02 00:03:47

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

v0.0.9
* Merge duplicated Software entries by App Name, Wiki Page and Timestamp
* Persist sorting settings
* Add more languages
* Update icon for Home Webpage button
* Handle more error cases
* Handle HTML codes in the description and name parameters
* Reorder platforms list

v0.0.8
* Refactoring.
* BUGFIX: wm2003 is interpreted as Desktop Windows version.

Offline

#23 2016-02-22 00:41:12

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

v0.1.1
* Add keyboard shortcuts
* Add Application Comparison dialog
* BUGFIX: FindNearbyPOI field displayed value of FindLocation field.

Last edited by Dmytro Ovdiienko (2016-02-23 00:38:57)

Offline

#24 2016-02-23 00:44:37

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

v0.1.3
Add filter by price (textual)
Add context menu to Compare Applications form

Offline

#25 2016-02-25 01:41:03

Dmytro Ovdiienko
Member
From: Kraków
Registered: 2010-02-15
Posts: 307

Re: Software catalog

v0.1.4
* Add option to filter by several Platforms
* Replace True/False with Yes/No
* Change colors in the Compare Application form (make it similar to Wiki)

Offline

Board footer

Powered by FluxBB