New stack: Difference between revisions

From KolibriOS wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
 
(142 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== intro ==
;This page describes the development of the new network stack for KolibriOS
; Since SVN #3556, the code has been merged in trunk and is now available in the english, russian spanish and italian builds of kolibrios!
;You can find the latest source code on svn://kolibrios.org/kernel/trunk ([http://websvn.kolibrios.org/listing.php?repname=Kolibri+OS&path=%2Fkernel%2Ftrunk%2F websvn])
;You can download an image file with pre-compiled library, applications, kernel and drivers on the [http://builds.kolibrios.org/ builds page]''<br />
;All bugs may be reported to [http://bugs.kolibrios.org KolibriOS bug tracker]


;As you may or may not know, I (hidnplayr) am since long time working on a new network stack for KolibriOS<br>
== Progress ==
;I will use this page for myself as a total image of what I have done so far, but also to inform others.


I'm currently working on:
System functions 52, 53 have been replaced by functions 74 (Network hardware access), 75 (BSD sockets) and 76 (Network settings and statistics). ''See sysfuncs.txt for details. <br />
* Complete rewrite of TCP code based on that of 4.4BSD
Internally, the network stack has been completely rewritten (mostly from scratch).<br />
All network drivers have been converted and or rewritten to operate with the new network stack.<br />


* Fragmenting IPv4 packets
*;Working:
* updating netstat and netcfg applications
**Attaching multiple network drivers/devices to the kernel
* Learning more about MII and transceiver chips
**Sending and receiving Ethernet/IPv4/UDP/ICMP/ARP/TCP packets
* Completing the dec21x4x driver
**Queuing of packets when necessary
* Completing/testing SLIP compatible RS232 driver
**UDP/TCP/RAW(ICMP/IP..) sockets (POSIX compatible)
* Finishing TFTP client
**Blocking sockets
**IPC/local sockets '' (needs testing)
**Dedicated network buffers '' (NET_buff_alloc, NET_buff_free)
**IP address assignment for all network interfaces


== Kernel ==
*;TODO:
 
**Link detection for all network drivers (Currently works on RTL8139, RTL8169, forcedeth, mtd80x, i8254x)
The old kernel functions 52 and 53 have been removed and the [[New_network_api]] has been created.
**Improved routing
 
**TCP: reassembly queue, testing (timestamps, RTT, PAWS, ...), ...
Internally, the stack has been completely rewritten (mostly from scratch)<br>
**PPP '' (need help developing/testing)
*;What works
**attaching network drivers
**sending and receiving Ethernet packets
**queuing of packets
**Sending and receiving of IPv4 packets
**Receiving Fragmented IPv4 packets
**Sending and receiving ICMP packets
**Sending and receiving UDP packets
**UDP sockets
**Working with multiple network card's at once (not fully tested)
**ARP code has been rewritten (altough not 100% bug-free yet)
**IP/UDP/TCP checksum generation and confirmation
**~ Establishing a TCP connection (active open)
 
*;TODO
**Sending fragmented IPv4 packets
**TCP
**IPC sockets
**RAW sockets (ethernet/ipv4/icmp/..)
**Enhance the API to work with network drivers
**Replace EthRegDev with more universal function
**Same for EthReceiver (order of registers on stack should change too)
**Port trunking (IEEE 802.1AX-2008)
**..


== Drivers ==
== Drivers ==


These new type network drivers are MS COFF files, just like the sound drivers in KolibriOS.<br>
In net-branch, the network drivers are external drivers, as described in [[Writing_drivers_for_KolibriOS]].<br />
;Notice that one driver can handle multiple devices.


Once the driver is loaded, an application (NetCFG) can use system function 68/17 ("Driver Control") to communicate with the driver.<br>
The specifications of the new network drivers can be found in the article [[Writing_network_drivers_for_KolibriOS]].<br />
For this, it uses the IOCTL structure wich looks like this:
<asm>
struc IOCTL {
  .handle      dd ?
  .io_code    dd ?
  .input      dd ?
  .inp_size    dd ?
  .output      dd ?
  .out_size    dd ? 
}
</asm>
for more info about this, please read [[Writing_drivers_for_KolibriOS]]


Here is a list of opcodes:
'''[[Hardware_Support#Network | Currently supported network hardware]]'''
;0 - Getversion
standard for all drivers, not only network drivers


;1 - Hook
'''Unsupported PCI card?''' If you have a PCI/PCI-X/PCI-express ethernet card which is not listed above, and would like to acces the internet in KolibriOS, you may send me the card, and I'll send you one back that is already supported.<br>
Attach driver to a device<br>


IN: (ISA)
== Programs ==
<asm>
type    db 0 ; isa
io_addr  dw ?
irq_line db ?
</asm>


IN: (PCI)
=== FTPc ===
<asm>
[[Image:ftpc.gif]]<br>
type    db 1 ; pci
CLI FTP client. <br />
pci_bus  db ?
''Uses console.obj''<br />
pci_dev  db ?
<br />
</asm>
Also see: [[Using kolibrios.org with ftpc]]


OUT:<br>
=== FTPd ===
eax = device number or -1 on error
Full fledged FTP daemon. Fully configureable through ini files. <br />
Status: Basic commands work. <br />
''uses libini.obj, libio.obj and console.obj, TODO: STOR command, dont use libio, ...''


=== IRCc ===
[[Image:Ircc.gif]]<br>
New IRC client. Under construction. <br />
''uses network.obj, libini.obj and box_lib.obj''


----
=== NetCFG ===
[[Image:netcfg.gif]]<br>
This program is used to load the network drivers.<br />
You can run it with parameter 'F' to silently load first detected card, or parameter 'A' to load all cards. <br />
No parameters will load the GUI.


=== Netstat ===
[[Image:ipv4.png]]<br>
This program allows you to read the status of ethernet, ipv4, icmp, arp udp and tcp.<br />
''TODO: allow user to set ARP settings, and save those to .ini file.''


Once the HOOK function of the driver has been called, the driver creates the NET_DEVICE structure, allocates some buffers needed for the driver, and calls the kernel function 'NetRegDev'.<br>
=== nslookup ===
This function registers a networkdevice (device, not driver!) to the kernel.<br>
Commandline DNS client.<br />  
It only needs one parameter: pointer to device structure (NET_DEVICE) in ebx.<br>
''This program uses network.obj and console.obj''
The function returns the device number in edi (..wich the driver then sends to the application in eax, see above..)


This NET_DEVICE structure is created and filled in by the driver, for every device it is hooked to.<br>
=== ping ===
The driver may use this to store pointers to buffers, descriptors,.. etc<br>
[[Image:ping.gif]]<br />
But the top of the structure must always look like this: (Since it is shared by the driver and the kernel)
A program to ping a remote host using the ICMP protocol<br />
Only fixed packet size for now.<br />
''uses network.obj, libini.obj and console.obj''


<asm>
=== SSH ===
NET_DEVICE:
[[Image:Ssh.png]]<br />
        .type          dd ?    ; Type field (ethernet/slip/...)
SSH client
        .mtu            dd ?    ; Maximal Transmission Unit (in bytes)
''uses network.obj, libcrash.obj and console.obj''
        .name          dd ?    ; Ptr to 0 terminated string


        .unload        dd ?    ; Ptrs to driver functions
=== SynergyC ===
        .reset          dd ?    ;
Software KM-switch, (Synergy - client) for windows/unix/.. versions, see http://synergy-foss.org <br />
        .transmit      dd ?    ;
This version is compatible with version 1.3.1<br />
Only mouse events are accepted in the current version.<br />
''There is a bug in the loading of libraries which might cause program to crash on loading''<br />
''uses network.obj, libini.obj and console.obj''


        .bytes_tx      dq ?    ; Statistics, updated by the driver
=== Telnet === 
        .bytes_rx      dq ?    ;
[[Image:telnet.gif]]<br>
        .packets_tx    dd ?    ;
A simple telnet client<br />
        .packets_rx    dd ?    ;
''This program uses network.obj and console.obj''


        .end:
=== TFTPc ===
</asm>
[[Image:tftpc.gif]]<br>
A TFTP client with a GUI. <br />
'' uses libio.obj, network.obj and box_lib.obj''


This is appended by a Type-specific structure, for ethernet this will be:
=== VNCclient ===
This program allows you to control a remote computer. The remote computer must be running a VNC server that support the RFB 3.3 protocol.<br>
The client currently supports RAW, CopyRect, RRE and TRLE encodings.<br>
Authentication to a password protected VNC server is possible through the "VNC Authentication" algorithm.<br>
There is currently no support for encryption.<br>
''TODO: ZRLE, ..''


<asm>
=== Zeroconf ===
        .set_mode      dd ?
This program reads settings from network.ini and configures the tcp/ip stack accordingly <br />
        .get_mode      dd ?
You may choose for a manual setup, or dhcp / link-local.<br />
''TODO: keep listening for DHCP release packets etc.''


        .set_MAC        dd ?
=== NetSurf ===
        .get_MAC        dd ?
Under construction <br />
''TODO: Improve front-end, ...''


        .mode          dd ?
=== SocketDBG ===
        .mac            dp ?   
Program that can be used for printing socket variables, usefull for debugging.
</asm>


Wich is then followed by the private data for the driver.
=== PPPoE ===
PPPoE dialer - unfinished


=== TCPserv ===
TCP server demo program, this program can be used with iperf client to measure performance of the stack.


When a packet is received by the driver, it copies it into a buffer allocated using 'KernelAlloc'.<br>
== Libraries ==
Then the driver pushes the size of the buffer and the pointer to the buffer onto the stack (in this particular order)<br>
And it calls 'EthReceiver' (or another procedure for another network device type).<br>
This kernel function will but the packet into the incoming queue, so the appropriate protocol handler can handle it later.
 
When the kernel needs to send a packet, it simple looks up the appropriate driver in it's driver table.<br>
In the table it will find the pointer to the device structure, so the driver can directly call the transmit function listed in the structure.
NOTE: the kernel will put the address of buffer to send in [esp+4], and length of that buffer in [esp+8]
 
----
 
== Current status of the drivers ==
 
*RTL8139: Working (FIXME: error occurs on wrapping of the RX buffer).
*SIS900: Working? (Written by clevermouse)
*3c90x/3c59x: Working (only boomerang for now) (FIXME: may hang on UpUnstall procedure)
*RTL8029: Working
*PCnet32: Working
*DEC21x4x: Working in Virtual PC, but not on real hardware.
 
*i8255x (Intel eepro 100): Port initiated (hidnplayr)
*mtd80x: Planned for future (hidnplayr)
*intel eepro 1000: Planned for future (hidnplayr)
 
== Programs ==
 
*;NetCFG: This is a program I wrote to load the drivers, it detects all network cards in your computer (PCI), and lets you load the appropriate driver for it.
You can run it with paramters 'F' to silently load first detected card, or parameter 'A' to load all cards
 
*;Netstat: This program allows you to read some variables from the stack.
 
*;ARPcfg: This program allows you to view the ARP entries, in future, you should be able to add and remove static entry's using this progam.


*;Zeroconf: This program is the newer version of what used to be 'autodhcp', it works with the new network API.
=== network.obj ===
 
Network library for KolibriOS, written by Clevermouse. Primary functions are getaddrinfo, inet_ntoa.
*;nslookup: Commandline DNS client from CleverMouse
  uses network.obj
 
*;TFTPc: A TFTP client with a GUI.
can receive and send files, using libio.obj, network.obj and box_lib.obj
 
*;Telnet:  a new telnet client, based on the code of nslookup (usefull to test TCP code)
  uses network.obj
 
== Upcoming ==
 
*;ICMP: program for sending ICMP echo reply's (ping) and trace-route.
 
== Libraries ==


*;network_lib: The network library for KolibriOS, written by Clevermouse, was ported to the net branch, by the same person
=== http.obj ===
HTTP client library, can be used to fetch content from the web. <br>
Currently support HTTP/1.1 GET, POST and HEAD methods. HTTP proxy's are also supported (settings in network.ini).


== Applications, old vs new ==


== I need your help! ==
{| class="wikitable" border="1"
|'''old'''
|'''new'''
|'''remarks'''
|-
|airc
|ircc
|IRC client (rewritten from scratch)
|-
|arpstat
|netstat
|ARP configuration utility (rewritten)
|-
|chess
|/
|may be ported
|-
|dhcp
|/
|replaced by zeroconf
|-
|dnsr
|nslookup
|Name service lookup (DNS resolver)
|-
|downloader
|downloader
|rewritten
|-
|ethstat
|/
|N/A
|-
|ftps
|ftpd
|FTP daemon (server)
|-
|https
|/
|may be ported, better to rewrite
|-
|icq
|/
|may be ported
|-
|ipc
|/
|may be ported
|-
|local
|/
|may be ported
|-
|mp3s
|/
|may be ported
|-
|netsendc
|/
|may be ported
|-
|netsends
|/
|may be ported
|-
|nntpc
|/
|may be ported
|-
|popc
|/
|may be ported (or replaced with liza?)
|-
|ppp
|/
|N/A (a new utility should be made)
|-
|rccc
|/
|may be ported
|-
|rccs
|/
|may be ported
|-
|remote
|/
|may be ported
|-
|smtps
|/
|may be ported
|-
|stackcfg
|netstat
|Network configuration utility (rewritten from scratch)
|-
|telnet
|telnet
|rewritten from scratch
|-
|terminal
|/
|N/A
|-
|tftpa
|/
|may be ported
|-
|tftpc
|tftpc
|Trivial File Transer Protocol Client
|-
|VNCclient
|vncc
|ported
|-
|ym
|/
|may be ported
|-
|zeroconf
|zeroconf
|ported
|}


I would really appreciate it if somebody could write some network applications,<br>
== We need your help! ==
so I can further test the kernel code without having to code them myself now :),<br>
;If you think you can help with further development of networking applications, driver or kernel code in any way, please let us know through the forum or IRC channel.  
If you are interessed please contact me at hidnplayr@kolibrios.org (or find me in #general on irc.kolibrios.org)


[[Category:Coding]]
[[Category:Coding]]
[[Category:System_documentation]]
[[Category:System_documentation]]

Latest revision as of 12:42, 10 July 2021

This page describes the development of the new network stack for KolibriOS
Since SVN #3556, the code has been merged in trunk and is now available in the english, russian spanish and italian builds of kolibrios!
You can find the latest source code on svn://kolibrios.org/kernel/trunk (websvn)
You can download an image file with pre-compiled library, applications, kernel and drivers on the builds page
All bugs may be reported to KolibriOS bug tracker

Progress

System functions 52, 53 have been replaced by functions 74 (Network hardware access), 75 (BSD sockets) and 76 (Network settings and statistics). See sysfuncs.txt for details.
Internally, the network stack has been completely rewritten (mostly from scratch).
All network drivers have been converted and or rewritten to operate with the new network stack.

  • Working
    • Attaching multiple network drivers/devices to the kernel
    • Sending and receiving Ethernet/IPv4/UDP/ICMP/ARP/TCP packets
    • Queuing of packets when necessary
    • UDP/TCP/RAW(ICMP/IP..) sockets (POSIX compatible)
    • Blocking sockets
    • IPC/local sockets (needs testing)
    • Dedicated network buffers (NET_buff_alloc, NET_buff_free)
    • IP address assignment for all network interfaces
  • TODO
    • Link detection for all network drivers (Currently works on RTL8139, RTL8169, forcedeth, mtd80x, i8254x)
    • Improved routing
    • TCP: reassembly queue, testing (timestamps, RTT, PAWS, ...), ...
    • PPP (need help developing/testing)

Drivers

In net-branch, the network drivers are external drivers, as described in Writing_drivers_for_KolibriOS.

The specifications of the new network drivers can be found in the article Writing_network_drivers_for_KolibriOS.

Currently supported network hardware

Unsupported PCI card? If you have a PCI/PCI-X/PCI-express ethernet card which is not listed above, and would like to acces the internet in KolibriOS, you may send me the card, and I'll send you one back that is already supported.

Programs

FTPc

Ftpc.gif
CLI FTP client.
Uses console.obj

Also see: Using kolibrios.org with ftpc

FTPd

Full fledged FTP daemon. Fully configureable through ini files.
Status: Basic commands work.
uses libini.obj, libio.obj and console.obj, TODO: STOR command, dont use libio, ...

IRCc

Ircc.gif
New IRC client. Under construction.
uses network.obj, libini.obj and box_lib.obj

NetCFG

Netcfg.gif
This program is used to load the network drivers.
You can run it with parameter 'F' to silently load first detected card, or parameter 'A' to load all cards.
No parameters will load the GUI.

Netstat

Ipv4.png
This program allows you to read the status of ethernet, ipv4, icmp, arp udp and tcp.
TODO: allow user to set ARP settings, and save those to .ini file.

nslookup

Commandline DNS client.
This program uses network.obj and console.obj

ping

Ping.gif
A program to ping a remote host using the ICMP protocol
Only fixed packet size for now.
uses network.obj, libini.obj and console.obj

SSH

Ssh.png
SSH client uses network.obj, libcrash.obj and console.obj

SynergyC

Software KM-switch, (Synergy - client) for windows/unix/.. versions, see http://synergy-foss.org
This version is compatible with version 1.3.1
Only mouse events are accepted in the current version.
There is a bug in the loading of libraries which might cause program to crash on loading
uses network.obj, libini.obj and console.obj

Telnet

Telnet.gif
A simple telnet client
This program uses network.obj and console.obj

TFTPc

Tftpc.gif
A TFTP client with a GUI.
uses libio.obj, network.obj and box_lib.obj

VNCclient

This program allows you to control a remote computer. The remote computer must be running a VNC server that support the RFB 3.3 protocol.
The client currently supports RAW, CopyRect, RRE and TRLE encodings.
Authentication to a password protected VNC server is possible through the "VNC Authentication" algorithm.
There is currently no support for encryption.
TODO: ZRLE, ..

Zeroconf

This program reads settings from network.ini and configures the tcp/ip stack accordingly
You may choose for a manual setup, or dhcp / link-local.
TODO: keep listening for DHCP release packets etc.

NetSurf

Under construction
TODO: Improve front-end, ...

SocketDBG

Program that can be used for printing socket variables, usefull for debugging.

PPPoE

PPPoE dialer - unfinished

TCPserv

TCP server demo program, this program can be used with iperf client to measure performance of the stack.

Libraries

network.obj

Network library for KolibriOS, written by Clevermouse. Primary functions are getaddrinfo, inet_ntoa.

http.obj

HTTP client library, can be used to fetch content from the web.
Currently support HTTP/1.1 GET, POST and HEAD methods. HTTP proxy's are also supported (settings in network.ini).

Applications, old vs new

old new remarks
airc ircc IRC client (rewritten from scratch)
arpstat netstat ARP configuration utility (rewritten)
chess / may be ported
dhcp / replaced by zeroconf
dnsr nslookup Name service lookup (DNS resolver)
downloader downloader rewritten
ethstat / N/A
ftps ftpd FTP daemon (server)
https / may be ported, better to rewrite
icq / may be ported
ipc / may be ported
local / may be ported
mp3s / may be ported
netsendc / may be ported
netsends / may be ported
nntpc / may be ported
popc / may be ported (or replaced with liza?)
ppp / N/A (a new utility should be made)
rccc / may be ported
rccs / may be ported
remote / may be ported
smtps / may be ported
stackcfg netstat Network configuration utility (rewritten from scratch)
telnet telnet rewritten from scratch
terminal / N/A
tftpa / may be ported
tftpc tftpc Trivial File Transer Protocol Client
VNCclient vncc ported
ym / may be ported
zeroconf zeroconf ported

We need your help!

If you think you can help with further development of networking applications, driver or kernel code in any way, please let us know through the forum or IRC channel.
Retrieved from "http://wiki.kolibrios.org/index.php?title=New_stack&oldid=5863"